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Executive summary 


Paired Reading is a peer tutoring programme in secondary schools which trains teachers to support 
and encourage the regular tutoring of Year 7 pupils (aged 11-12 years) by Year 9 pupils (aged 13-14 
years). The Paired Reading programme aims to improve pupils’ general literacy in addition to speaking 
and listening skills. This is achieved by pupils working together to follow the Paired Reading steps to 
choose the material to read, and discuss this, together with the older pupil (tutor) supporting the reading, 
correcting errors and praising the younger pupil (tutee) throughout. 


The 16-week programme is intended to take place during normal school hours in timetabled sessions, 
for 20 minutes each week. Teachers in participating schools received training from the delivery team, a 
detailed programme manual and extensive digital resources. 


The impact of Paired Reading on 2,736 pupils in 120 classes in ten participating schools (1,370 in Year 
7 and 1,366 in Year 9) was tested using a cluster randomised controlled trial design with 58 classes 
randomly allocated to receive the programme and 62 classes allocated to the control condition. Schools 
from the North Tyneside local authority (LA) and in neighbouring LAs of South Tyneside and Sunderland 
took part in the trial over the 2013/14 academic year. 


Key Conclusions 


This evaluation does not provide any evidence that the Paired Reading programme had an 
impact on overall reading ability, sentence completion and passage comprehension of 
participating pupils. 

There was no evidence of the Paired Reading programme having an effect on overall reading 


ability, sentence completion and passage comprehension of FSM pupils. 

There was some variation in the intervention group schools in terms of the programme set- 
up and delivery. There was also a varying level of support provided to pupils within the 
intervention by the teachers involved, based mainly on the reading ability of the pupils. 
However, these appear to be natural variations between the settings of the schools involved 
and are unlikely to have affected the dosage of the intervention for the pupils involved. 


What impact did it have? 


There is no evidence from this evaluation that the Paired Reading programme had any positive impact 
on overall reading ability of participating pupils, measured using the New Group Reading Test (NGRT) 
scores (see key findings for the primary outcome, reading ability, below). 


The process evaluation suggested that there were some differences amongst participating schools with 
regards to the set-up, delivery and implementation of the programme. There was also a varying level 
of support provided to pupils within the intervention by the teachers involved, based mainly on the 
reading ability of the pupils. 


Security rating awarded as part of 
Security rating the EEF peer review process 


Findings from this evaluation have moderate to high security. The study was set up as a randomised 
controlled trial, which aimed to compare the progress of Year 7 and 9 pupils who received the 
programme to similar pupils who did not. The trial was classified as an effectiveness trial, meaning that 
it sought to test the intervention in realistic conditions in a large number of settings. 


The study was a large and well-conducted study. All classes allocated to intervention or control arms 
completed the study. As the randomisation took place at class level this does introduce the possibility 
of an exchange of information between teachers and pupils in different trial groups. The process 
evaluation did not indicate information was exchanged. 
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At the end of the intervention period all pupils were asked to complete the New Group Reading Test, 
as a Standardised measure of general reading ability. As the testing was administered by the schools 
the padlock rating is reduced to 4. 


How much does it cost? 


The cost of the Paired Reading programme is estimated to be £10.50 per pupil. This estimate includes 
training and support costs and materials required for schools to be able to deliver the programme. These 
estimates are based on a class of 25 pupils (and five classes per school). 


Group Effect size Estimated Security 
months’ rating 
Progress 


“1° 8888 


Intervention vs. 
control (Year 7 


Intervention vs. 
=i &a@88 


control (Year 9 
Free school 
meal pupils 


Free school 
meal pupils 
(Year 9) 


1 Since this report was published, the conversion from effect size into months of additional progress has been 
slightly revised. If this result was reported using the new conversion, it would be reported as 0 months of additional 
progress rather than -1. See here for more details. 


2 See footnote 1. 
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Introduction 


Intervention 


Paired Reading is a Peer Tutoring programme in secondary schools which trains teachers to support 
and encourage regular tutoring of Year 7 pupils (aged 11-12 years) by Year 9 pupils (aged 13-14 years). 
The Paired Reading programme aims to improve pupils’ general literacy in addition to speaking and 
listening skills. This is achieved by pupils working together to follow the Paired Reading steps to choose 
the material to read, and discuss this together with the older pupil (tutor) supporting the reading, 
correcting errors and praising the younger pupil (tutee) throughout. 


In Paired Reading, pupils in Year 7 are paired with pupils in Year 9 using recent reading assessment 
scores and spend 20 minutes each week reading together, over 16 weeks. Pupils are trained and 
supported by their teachers to follow clear procedures for each session. Pupils select reading materials 
(e.g. books, magazines) above the independent reading ability of the tutee, then spend a short period 
reading together before the tutee begins to read aloud on their own. The tutor then focuses on error 
correction. Each time the tutee makes a mistake the tutor will wait four or five seconds before intervening 
with the correct word. In addition this Paired Reading process includes a meta-cognitive element 
developed through the use of questioning and feedback by the pair. 


The Paired Reading programme is used in conjunction with the school’s English curriculum. It is not 
intended to replace other English/literacy teaching but rather is incorporated into lessons to support 
pupil’s reading development. Teachers in participating schools received training from the delivery team, 
a detailed programme manual and extensive digital resources (Thurston & Cockerill, 2013). 


Schools from North Tyneside local authority (LA) in North East England and in neighbouring LAs of 
South Tyneside and Sunderland took part in the trial. 


Background evidence 


Peer tutoring is a student-led instructional strategy used to support improved academic achievement 
and social-emotional outcomes of pupils. Research suggests that pupils involved in peer tutoring show 
higher academic achievement, improved relationships with peers, improved personal and social 
development and increased motivation. The evidence for peer tutoring tends to be overwhelmingly 
positive. For example, a meta-analytic review summarising the findings from 65 independent 
evaluations of school tutoring programmes by Cohen, Kulik & Kulik (1982) showed that these 
programmes have positive effects on the academic performance and attitudes of those who receive 
tutoring but also on tutors. A more recent review by Rohrbeck and colleagues (2003) looking at 
comparison design studies evaluating peer-assisted learning interventions with elementary school 
students showed that such interventions increased pupils’ academic achievement. The review also 
indicated that interventions that used interdependent reward contingencies, ipsative evaluation 
procedures, and provided students with more autonomy showed more positive effects. 


A recent meta-analytic review by the Washington State Institute for Public Policy (WSIPP) (Pennucci & 
Lemon, 2014) explored the effects of both cross-age and same-age/class wide peer tutoring.? The 


3 In the two evaluations included in the meta-analysis of cross-age peer tutoring, the average cross-age 
peer tutoring programme provides 30 hours tutoring time and 7.5 hours of training time per class. In the 
evaluations included in the meta-analysis of same-age peer tutoring, pupils from the same classrooms 
provide academic assistance to struggling peers. Same-age tutoring assistance occurs through one- 
on-one interactions or in small groups, and in some instances, students alternate between the role of 
tutor and tutee. The same-age peer tutoring programmes included in the WSIPP’s meta-analysis 
provide, on average, 30 hours of peer tutoring time each year and about five hours of training time for 
teachers and students to learn programme procedures. 
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results of respective meta-analytic reviews indicated that both types of programmes had a positive effect 
on participant test scores. However, the strength of the evidence was found to be stronger for same- 
age and class wide peer tutoring when compared cross-age peer tutoring in the WSIPP inventory of 
evidence- and research-based practices. The meta-analytic review on cross-age peer tutoring included 
only two relatively old studies (1980 and 1983) that included limited information on the demographic 
characteristics of the pupils (for further details on the criteria applied to the included studies please see 
Pennucci & Lemon, 2014%). 


Poor literacy outcomes continue to affect many young people in England, including at the transition 
between Primary and Secondary school and this is most acute for pupils living in areas of multiple 
disadvantage. Gaps in attainment widen between pupils known to be eligible for Free School Meals 
(FSM) and their peers through Key Stage 2, and persist during Key Stage 3 at secondary school (DfE, 
2014a). Whilst reading tends to show smaller differences in progress by FSM status, this is still of 
concern to policy makers and practitioners (DfE, 2014a). 


The EEF and Sutton Trust Pupil Premium Toolkit (Higgins, Katsipataki, Coleman et al., 2014)) included 
a review of the effectiveness of different teaching and learning strategies and concluded that based on 
current evidence peer tutoring was a potentially effective approach to improve attainment for both the 
tutors and tutees — who have to think carefully about how to support the tutees. The evidence of impact 
is relatively high, typically equating to about a GCSE grade, and there is some evidence that children 
from disadvantaged backgrounds and low attaining pupils make the biggest gains. 


A large-scale efficacy trial in Fife (Tymms, et al., 2011) developed the peer tutoring approach in primary 
school maths and reading using the Paired Reading technique for the latter. The study showed that 
cross-age peer tutoring had a consistent positive impact on attainment in both math and reading for 
both the younger and older cohorts (and produced better results when compared to same-age tutoring). 
The study had limitations including researchers not being blind to treatment allocation, loss of schools 
and pupils during the project, schools changing their intervention group and a differential loss to follow 
up among the older cohort. Despite limitations, this study suggested that peer tutoring has promise 
when scaled up. 


Recently, a research team from Durham University modified the existing ‘Fife’ Cross-Age Peer Tutoring 
model for maths. The impact of this cross-age peer tutoring pedagogy called Durham Shared Maths 
was tested in 82 primary schools in four English local authorities to assess how effective it is at raising 
the attainment of disadvantaged pupils when implemented on a large scale and delivered by non- 
specialists in peer tutoring. This effectiveness trial, however, did not provide evidence of an impact on 
attainment in maths as the primary outcome (Lloyd, et al., in review). The results of this trial are 
expected to be published in May 2015. 


This EEF effectiveness trial of Paired Reading builds on the Fife experience by delivering the 
programme to secondary school pupils in the North East of England to assess how effective it is at 
raising the literacy attainment of disadvantaged pupils, shortly after the transition to secondary 
education. 


Evaluation objectives 


The aim of the impact evaluation was to measure the impact of the Paired Reading programme on the 
reading ability attainment of participating pupils: 


e in Year 7 (the tutee) 


4 For combined meta-analytic results on class-wide peer tutoring, peer-assisted learning strategies and 
reciprocal peer tutoring by WSIPP please visit the following link: 
http://www.wsipp.wa.gov/BenefitCost/Program/107 
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e in Year 9 (the tutor) 

e who are eligible for FSM in both tutee and tutor year groups 

e who are in the special educational needs (SEN) category in both tutee and tutor year groups 
e who have English as an additional language (EAL) in both tutee and tutor year groups 

e by looking separately at girls and boys in both tutee and tutor year groups 


The aims of the process evaluation were to: 


e Understand, at a strategic level, what attracted schools to sign up to the programme. 

e Understand the process of setting up the programme in the school- including pairing the pupils 
and finding space in the timetable. 

e Explore how the programme worked in practice and any implementation, fidelity or sustainability 
issues. 

e Find out how teachers and senior leaders in schools perceived the programme to be making 
(or not making) an impact — whether on the pupils or on the school at a wider level. 

e Examine how the teachers who delivered the programme in the schools felt the programme 
could be improved if it were to be implemented more widely. 

e Look into issues around sustainability of the programme. 


Project team 


The programme was implemented by a delivery team led by Maria Cockerill of North Tyneside Council, 
in partnership with Allen Thurston and Nicole Craig of Queen’s University, Belfast, with design 
collaboration from Christine Merrell of Durham University. 


The independent evaluation was set up by a team from the University of Bristol, led by Paul Clark who 
managed the project through set up and randomisation for the trial. However, Paul Clark left the 
University of Bristol and the EEF re-tendered for the evaluation work. For the remaining stages of the 
project a team from NatCen Social Research were the independent evaluators. The impact evaluation 
was led by Cheryl Lloyd who was assisted by Triin Edovald, Zsolt Kiss and Stephen Morris. The process 
evaluation was designed and overseen by Amy Skipp, assisted by Hashim Ahmed. 


Ethical review 


NatCen Social Research obtained ethical approval from its own ethics board for the evaluation, 
comprising the process evaluation and analysis of test results. This approval included the processes 
for the research team communicating with and carrying out interviews with school staff. 


This trial has been registered at www.controlled-trials.com with the trial number ISRCTN03012788. 
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Methodology 


Trial design 


In order to assess the impact of the Paired Reading programme on the reading ability of participating 
children, a cluster randomised controlled trial with assignment at class level was carried out in ten 
participating secondary schools. Classes at these schools were assigned at random into two trial groups 
— intervention and control (See Randomisation section below for more details). Pupils in intervention 
classes participated in Paired Reading while pupils in control classes received ‘business as usual’ 
(BAU). Thus, pupils in the control group continued to operate as usual. Schools participated in the 
project from January to June 2014. 


The design study is a pragmatic randomised controlled trial. Namely, the schools participating in the 
study were the ones whose head teachers had volunteered to be part of the trial and that included 
classes where the logistics of the standard school day allowed the implementation of peer tutoring. 


Eligibility 


Ten secondary schools in North East England were recruited to take part in the trial, selecting schools 
where the proportion of pupils eligible for free school meals (FSM) was above the national average. 
Further details on recruitment are presented in the section on Participants below. All Year 7 and Year 
9 classes in recruited schools were eligible to participate in the study. However, due to the nature of the 
Paired Reading programme, an equal number of Year 7 (tutee) and Year 9 (tutor) classes of similar 
size were required in each participating school. Therefore, some of the classes in Year 7 or Year 9 
group were excluded from the trial. For further details see Randomisation section below. 


The head teachers at participating schools signed a Memorandum of Understanding (MoU) with the 
delivery and evaluation teams, agreeing that they consented on behalf of their school to randomisation, 
pupil testing and running the Paired Reading programme. Following this agreement, each school sent 
a letter to the parents of all pupils in Year 7 and 9 which was provided by the delivery team. This letter 
explained that the school was participating in the Paired Reading programme, including details of the 
intervention, pre- and post-testing and asked parents to inform the school if they did not wish their child 
to take part (see Appendix A for the parent Opt out letter). 


Intervention 


The Paired Reading programme is a cross-age peer tutoring programme in secondary schools which 
involves older Year 9 pupils tutoring younger Year 7 pupils. The participating children work together to 
follow the Paired Reading steps to choose the material to read which can be a book, magazine or any 
other written document, and discuss this together with the older pupil (tutor) supporting the reading, 
correcting errors and praising the tutee throughout. Paired Reading has been developed from a number 
of earlier peer tutoring projects, originating from Keith Topping and colleagues’ work at Dundee 
University. 


The Paired Reading programme aims to improve pupils’ general literacy in addition to speaking and 
listening skills. This is achieved by the tutor and tutee working together to read text aloud, with the older 
pupil correcting errors and the two pupils discussing the content of the text to deepen their 
understanding of the issues raised and critically thinking about these which aims to encourage reading. 
The two pupils are required to listen to each other, with the older pupil praising the tutee as appropriate 
throughout which develops speaking and listening skills (Thurston & Cockerill, 2013). 


The programme follows the steps: 
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1. The younger pupil, who is acting as the tutee, chooses a book which is above their independent 
reading ability. 

2. The younger and older pupils spend a short time reading the book aloud together. 

3. The older pupil then stops reading aloud while the younger pupil continues reading on their 
own. 

4. The older pupil, who is acting as the tutor, will then focus on error correction. 

5. When the tutee struggles to or cannot pronounce a word, the tutor waits for five seconds before 
intervening with the correct pronunciation. 

6. Both the tutee and tutor begin reading together again. 

7. In addition to correcting pronunciation, the tutor and tutee will question one another, before, 
during and after the reading. The questions could be comprehension based and focused on the 
book that the tutee has chosen to read. 

8. The tutor and tutee provide ongoing assessment through feedback using verbal feedback 
recorded in a structured log book. 


This provides a formal system of ongoing peer assessment through feedback. 


The Paired Reading delivery team were responsible for providing teacher training for both Year 7 and 
Year 9 teachers and on-going support throughout the programme. (Further details on both teacher and 
pupil training can be found below.) 


Prior to the start of the programme, teachers in participating schools received training from the delivery 
team and a detailed programme manual. The Peer Tutoring in Secondary Schools manual (Thurston & 
Cockerill, 2013) provided background information about the programme, how this links to the National 
Curriculum and how it should be delivered to pupils in their school. (Please see Process evaluation — 
Role of the teacher for further details.). 


As set out in the manual, the Paired Reading programme is intended to take place during normal school 
hours in timetabled sessions. Pupils in Year 7 are paired with pupils in Year 9 using recent reading 
assessment scores and spend 20 minutes each week reading together, over 16 weeks. Both tutors and 
tutees are trained and supported by their teachers to follow clear procedures for each session. 


Teacher training 


Prior to the start of the programme, teachers from participating schools received a day of training, 
covering the rationale and background to the intervention, as well as the practical aspects of delivering 
the programme on a day to day basis. The training covered the processes for training pupils, pairing 
pupils, selecting books for reading, supported reading, error correction, questioning and praise. 
Attendees were provided with a manual advising them on techniques and the approach. A further % 
day, was provided four weeks into the programme for teachers to further improve their practice and to 
consolidate teacher school teams for the duration of the programme. Further materials were provided 
during the second training event in respect of enhancing communication skills between peer tutors and 
tutee pupils. In most cases Year 7 and Year 9 teachers attended the training. However, if another 
subject teacher was responsible for delivering the Paired Reading programme in the school, this teacher 
attended instead. Following the training, participating schools were provided with a resource pack 
containing the training slides, other background information about the intervention and other practical 
materials required, such as log books and question mats. In addition, teachers were provided with 
access to a website containing all online resources, including training films and teacher and student 
training PowerPoint resources for sessions 1-16. 


Contact time 


It was recommended that schools set aside 30-minute periods for the programme with 20 minutes of 
this dedicated to quality reading time. The other 10 minutes were planned to provide time for pupils to 
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move between classrooms and getting them settled for reading. Schools were advised to conduct the 
intervention during normal school hours, replacing another lesson in the pupils’ timetable. One school 
was unable to do so due to timetable constraints and, instead, brought pupils into school early in order 
to conduct the intervention, replacing the normal registration period. 


The intervention was designed to be used at any time during the school day (not specifically during 
English lesson classes). 52 classes delivered the intervention during normal school hours and 6 classes 
delivered half of every session before the school day and the other half at the start of the day - 
consecutively. Of the 52 classes, 20 received the intervention during English class time. 


Pupil training 


The Paired Reading manual recommends that schools train both tutors and tutees together to ensure 
that both received the same information about the Paired Reading programme. The programme manual 
advises that the first four sessions should be used as follows to train pupils and ensure all pairs were 
confident with how the programme works: 


1. The first session for explaining the background and rationale as well as any starter activities 
with the pairs working together. 

2. The second session for practising praise and error correction. 

3. The third session to introduce questioning. 

4. The fourth session to correct any areas of weakness and consolidate learning to date. 


Pupil training took place either with Year 9s and Year 7s together or separately. PowerPoint resources 
were provided for the 16 sessions, with separate distinct PowerPoint slideshows for sessions 1 to 4 to 
incorporate the training element for pupils and a further generic PowerPoint for sessions 5-16. There 
was some variation across schools in terms of how the training took place and in terms of the training 
length; for example, some schools used only the first lesson to train the pupils, while others provided a 
staged introduction to the programme. Some schools only trained Year 9 pupils and there was an 
incident where one school did not carry out any advance pupil training (for further details see Process 
Evaluation - Training the pupils section below). 


Pairing classes and pupils 


Year 7 and Year 9 classes in the intervention group were paired up based on reading ability, using the 
pre-intervention test scores [Standard Age Scores from the digital New Group Reading Test (NGRT)].° 
The evaluation team at NatCen were responsible for recommending which pairs of classes from each 
Year group should work together for the Paired Reading programme. This was done by ranking classes 
within each school according to their average test scores, each class in Year 7 was then matched with 
the same ranking class in Year 9. These matches were then checked to ensure that the demographic 
characteristics of pupils were similar in terms of gender, FSM, EAL and ethnicity. 


Teachers were responsible for pairing pupils from Years 7 and 9 within paired classes assigned to the 
intervention group. Pupils were ranked in the order of reading levels based on their pre-intervention test 
scores (See Outcomes section for more details about the tests). Pupils in each pair of Year 7 and 9 
classes were then paired with one another, with the top pupil in Year 7 paired with the top pupil in Year 


5 The preferred option of pairing classes and individual pupils is based on a recent reading test to enable 
the best match between Year 9 tutors and Year 7 tutees in respect of reading age. For the purposes of 
this project the pre-intervention test score was used, however, a specific assessment is not required for 
the Paired Reading programme so another appropriate, recent measure of reading ability would 
normally be used. 
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9, the second pupil in Year 7 paired with the second pupil in Year 9, and so on. Other than reading 
levels, schools were advised to consider other factors when pairing pupils, such as gender, maturity, 
working habits, cultural background, personality and attendance. 


If there was an uneven number of pupils in Year 7 and 9, then some pupils were allocated into ‘trios’. 
The trios could either be with two tutors and a tutee or two tutees and a tutor. Poor attendance was 
recommended as a factor for inclusion into a trio, to ensure there was at least an opportunity for pupils 
to read in pairs during every session (even in the absence of a member of a trio). 


Training the teachers 


The training for participating staff was delivered at a local council run education provider centre. It was 
delivered over two separate sessions, totalling one and a half days for each participating teacher. 
Session 1 lasted a full day and Session 2 was a follow up two to three weeks into the programme 
delivery, which lasted half a day. The teachers who would be delivering the intervention, from all 
participating schools, received the training but not all at the same time; rather they attended separate 
sessions in order to minimise the number of teachers away from the school at once. Teachers 
commented that receiving the training along with teachers from other schools was a positive aspect as 
it helped to widen their professional understanding and offered the opportunity for them to learn from 
others. 


‘It is always good to meet other teachers and learn from them and their experiences’ 
(Teacher) 


The format of the training was that teachers: 


e Were given an introduction to the programme, including the rationale behind the intervention 
e Went through how the programme works and what is expected of the teachers and pupils 
e Were put in pairs and practiced the programme as if they were pupils receiving the intervention. 


All participants were given a support pack that contained resources and materials explaining how to 
deliver the programme. This included a copy of the presentation slides and the materials needed for 
the programme; such as question cards, question mats and log books. In addition, teachers were given 
access to all resources, including films and detailed PowerPoint presentations for sessions 1-16 of the 
programme. There was ongoing technology support available from the delivery team to ensure all online 
resources were accessed by teachers, and where teachers required support the information was sent 
via email to anyone who requested it. 


Teachers spoke very positively about the training, in particular its more practical aspects. The practice 
activities were felt to provide a deeper understanding of the programme and gave teachers a clearer 
idea of what it would involve when implemented in their classrooms. 


‘That (the practice activity) was helpful because it made it a lot easier to explain it to our kids’ 
(Teacher) 


It was also felt that the training was complimented well by the support pack. The materials provided 
were felt to cover the practical issues teachers were likely to encounter as well as setting out what the 
sessions should involve and answering any outstanding questions they had about the process. 


‘Once | received my pack with all the information, | was pretty happy that | knew what to do with it all’ 
(Teacher) 
Teachers appreciated the ‘hands-on’ format of the training as they felt it helped them get experience of 


the approach required, before getting their pupils to carry it out. This helped them to be better prepared 
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to deliver the programme once it began. They practiced the programme step by step, including selecting 
a book, reading together, correcting, questioning and praising. 


It wasn’t just sitting and watching someone read from a PowerPoint...there was a chance to have a go 
at some of the things that were part of the programme’ 

(Teacher) 
However, there were suggestions for how the training could be improved further: 


e Some of the resources contained grammatical errors, which did not create a good impression. 

e Some teachers experienced difficulties with electronic materials and downloading these from a 
dedicated website. Due to these difficulties some of the schools ended up creating their own 
resources, rather than rely on the ones provided. However these problems were not reported 
elsewhere and were by no means the prevailing experience. 


‘The whole thing could have been sent out by email. The whole scheme has been really time consuming’ 
(Teacher) 
e The training could have been reduced into a day, or even half a day, or delivered as 
departmental time within their schools. There was concern about the amount of time the 
programme delivery took up overall and reducing the training length would have helped to 
alleviate this. 


Selecting reading text 


Schools were instructed to allow the tutee to select their reading material. Teachers were encouraged 
to help pairs try different genres of text and ensure the books were at the right level of reading difficulty. 
Pupils tended to select a book and continue reading the book until it was finished, at which point they 
would select a new one. 


The ‘Five Finger Test’ illustrated below (Source: Thurston & Cockerill, 2013) was recommended to select 
a book. Ideally, the book would be above the independent reading ability of the tutee but below that of 
the tutor to facilitate the tutor being able to help the tutee as far as possible. 
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Five Finger Test 


Choosing ‘Just Right’ books 
Use the Five Finger Test 


¢ Tutor or tutee: put 5 digits on the page 
¢ Tutee: read the words under each digit 
¢ Repeat the process over four pages 


Add up the total number of mistakes: 


¢1 or less = book is too easy 
¢2 - 3 = book is ‘Just Right 
¢ 4 or more = book is too hard 


Paired Reading process 


Once books have been selected the approach then encourages pupils to follow the steps set out in 
detail below (Thurston & Cockerill, 2013): 


1. Supported reading 
The pairs begin to read together, with the tutor helping the tutee by modulating the speed of their reading 


to be just behind the tutee. After a short while, the tutor stops reading with the tutee continuing to read 
alone until a mistake was made. At that point, the pair would begin reading together again for another 
short while. 


2. Error correction 
When a tutee makes an error in pronunciation, the tutor waits for 4 or 5 seconds, providing the tutee 
with the opportunity to self-correct. Should this self-correction not occur, the tutor then says aloud the 
correct pronunciation of the word, which the tutee repeats. The tutor then gives praise and the pair 
starts reading together again. 


3. Questioning 
Questioning is considered to be one of the programme's key aspects as it meant both tutors and tutees 
needed to think about and understand the book they were reading. The questioning involves both tutors 
and tutees asking each other questions about the book before, during and after the reading. Question 
mats with suggested questions are provided to help with this, although pupils were also encouraged to 
think of their own questions, particularly as the intervention progressed. The questions generally 
covered: 


Before the reading 


e The reasons for choosing the book 
e = What will happen in the book 
e Whether they think they will like the book 
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During the reading 


e Who the favourite characters are 
e What the book is about 
e What is likely to happen next 


After the reading 


e Whether they enjoyed the story and why 
e Whether they would recommend it to others 
e Whether the story ended as expected 


The question mats provide questions of varying difficulty so that higher ability pupils can ask more 
difficult questions while the lower ability pupils ask questions that are easier for them to understand. 
There were three levels of question mats provided: 


Basic 
Standard 
Advanced (1 and 2) 


Teachers were instructed to use their own judgement to decide which mat to provide each pair with at 
the start of the paired reading project. They would then, whilst observing the pairs, make a decision 
about when to move them onto a more (or indeed less) advanced question mat as the programme 
progressed. 


4. Praise 
Tutors are encouraged to praise the tutees, for example when the tutee reads a difficult word or 


sentence, when the tutee pronounces a corrected word and when the tutee has read a section on their 
own. Tutors are provided with a praise card with words such as ‘good reading’ and ‘brilliant expression’ 
written on them and are encouraged to use these, along with their own praise words. 


5. Log books 
Both tutors and tutees are provided with log books in which to write after each session. The log allows 


pupils to note how far they had read into the book with space for both tutees and tutors to write how the 
tutee’s reading has improved from the previous session. 


Control classes 


The control group classes followed ‘business as usual’ during the trial and did not receive any Paired 
Reading materials or training. ‘Business as usual’ included interventions such as Literacy Clubs, 
additional reading support groups, withdrawal groups, specific ‘Literacy’ lessons and dedicated reading 
time (please see Other literacy and reading interventions in the Process evaluation section for further 
details on ‘business as usual’). 


Outcomes 
Primary and secondary outcomes 


The New Group Reading Test (NGRT), which was developed by GL Assessment and the National 
Foundation for Educational Research (NFER) was used to measure the outcomes of interest. The 
primary outcome of this trial was reading ability that was measured using the Overall Reading Scale of 
the NGRT that combines two NGRT subscales: the Sentence Completion Scale and the Passage 
Comprehension Scale. These two subscales of NGRT were used to measure the two secondary 
outcomes: sentence completion and passage comprehension. 
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The NGRT was administered by computer, using a computer adaptive version which assesses pupil 
performance during the test and adapts the questions asked to be appropriate for the pupil’s reading 
level. 


Pupil responses from each of these modules were used to calculate scale scores for sentence 
completion, passage comprehension and overall reading ability. Furthermore, Standard Age Scores 
(SAS) score was calculated for reading ability.6 The SAS is based on the pupil's raw test score which 
has been adjusted for age and placed on a scale that makes a comparison with a nationally 
representative sample of students of the same age across the UK. The average score is 100, with pupils 
scoring between 90 and 100 being considered ‘average’, those achieving scores of over 100 being 
‘above average’ and those scoring under 90 performing ‘below average’. 


For further details on specific scales and scores used in the analysis please see the Analysis section 
below. 


Administration of the NGRT 


All Year 7 and Year 9 pupils in participating schools undertook the pre-test in September 2013 and the 
follow-up test in June 2014. The NGRT was completed online and administered to pupils in groups or 
as a class, in exam conditions. 


The tests were administered by schools with some support from the Paired Reading delivery team in 
respect of technical set-up and not assistance with test completion. Teachers and the Paired Reading 
delivery team were aware of the pupils’ group allocation. As the computer administered version of the 
NGRT is an adaptive assessment, tailored to each individual pupil according to their abilities, the 
assessment marking is automated, and thus blinded to pupils treatment allocation status. 


Prior to test administration, the staff member responsible for Information Technology (IT) at each school 
uploaded pupil Unique Pupil Numbers (UPN) and background information to the NGRT GL Assessment 
website. Then, shortly after testing was complete the delivery team were able to securely access test 
scores and pupil background information. Once all testing was complete the data were provided to the 
evaluation team for matching to the National Pupil Database (NPD) and analysis. 


Sample size 


The sample size calculations that were used to inform the size of the trial were conducted by Bristol 
University before NatCen were involved in the project. NatCen have been unable to adequately 
ascertain the parameters and the design characteristics that were used in the calculation. So for the 
sake of clarity we have conducted a post-hoc power calculation for the number of schools and pupils 
actually recruited. 


Following the collection and processing of trial data, minimum detectable effect sizes (MDESs) for the 
primary outcome were calculated more precisely, using actual values for the ICC obtained from an 
unadjusted analysis’ (see below). The ICC for Year 7 data were generally higher than those obtained 
from Year 9. The proportion of variance explained at Level 2 — the class — subsequent to including 
covariates in the adjusted analysis was very high at 0.90, and at Level 1 - the pupil - around 0.60. 
MDESs at randomisation and as analysed are the same for both Year 7 and Year 9, 13 per cent of one 
standard deviation, and 11 per cent respectively. 


6 http:/Awww.gl-assessment.co.uk/sites/gl/files/images/Files/NGRTsub-scalepaper.pdf 
’ The unadjusted primary analysis includes all pupils who had post-test outcome data. 
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Table 1: MDESs for Year 7 and Year 9 at randomisation and at analysis 


As randomised As analysed 
Year 7 Year 9 Year 7 Year 9 
Classes N=60 Classes N=60 Classes N=60 Classes N=60 


Probability level 


Average cluster 23 


size 

R-squared (Level 2 
variance 
explained) 


Randomisation 


All Year 7 and Year 9 classes in participating schools were eligible for randomisation. However, as 
indicated above, an equal number of Year 7 (tutee) and Year 9 (tutor) classes of similar size were 
required in each participating school to deliver the Paired Reading programme. In order to achieve an 
equal or an approximately equal number of pupils (within classes) in both year groups included in the 
trial, all classes were stratified by a year group (Year 7 and Year 9), and listed alphabetically (e.g. 7a, 
7b, 9a, 9b) or numerically (e.g. 7/1, 7/2, 9/1, 9/2). The year group with the smaller number of classes in 
each school set the limit of classes in both year groups to be included in randomisation. Any additional 
classes in either Year 7 or Year 9 group were excluded before randomisation. 


In one of the participating schools pupils were separated by academic ability into English classes. More 
specifically, classes were streamed in 5 classes in Year 7 and 4 classes in Year 9. In both year groups, 
one class was half the size of the other classes and composed of the lowest ability pupils. Year 7 had 
four classes of similar size (1, 2, 3, 4) with classes 2, 3, 4 being of similar ability and class 5 with smaller 
numbers in the lowest ability set. Year 9 had three classes of similar size (1, 2, 3), with classes 2 and 3 
being of similar ability, and class 4 with a smaller number of pupils showing lowest ability. In this school, 
although the classes in each year group were first listed as described above, Year 7 English class 4 
was excluded pre-randomisation to ensure that the four possible streamed classes in each year group 
were of similar size. This was part of the trial design to ensure a similar number of pupils would be 
paired between classes as tutors or tutees. 


Randomisation was conducted as follows. Once an even number of Year 7 and Year 9 classes in each 
school was achieved, two groups were created, group A and group B, by going through the list of Year 
7 and Year 9 classes in each school and alternately assigning a class first to group A and then group 
B. The classes in group A and group B were then randomly assigned to either intervention or control 
group. A member of the Bristol University evaluation team carried out the randomisation procedure by 
tossing a coin. When the randomisation was conducted, ‘heads’ was identified as intervention and ‘tails’ 
as control. As the result was ‘tails’, the first class available on the list was allocated to the control group. 
From then all subsequent available classes were allocated alternately to control group or intervention 
group. The evaluator carrying out the randomisation procedure declared the outcome over the phone 
to the delivery team. 


See section Pairing classes and pupils above for more details on how classes and pupils were paired 
post-randomisation. 


Analysis 


The analysis was conducted in STATA version 13 (Stata Corporation, College Station, Texas, USA). 
Impacts were estimated on the basis of intention to treat, whereby all classes and pupils were analysed 
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according to the study arm to which they were initially assigned, regardless of whether they went on to 
participate in the intervention. 


An overview of the study outcomes, the NGRT scales used to measure the outcomes and various 
scores used in the analysis is provided in Table 2. 


Table 2: An overview of primary and secondary outcomes and outcome indicators used in the 
analysis 


Outcome Measure’ Scale Score used Score usedin Score used in 


Reading ability 
(primary) 


Sentence 
completion 


(secondary) 


Passage 


The Overall 


Reading Scale 


The Sentence 
Completion 
Scale 

The Passage 


in 
unadjusted 
descriptive 
analysis 

Scale score* 


Scale score* 


Scale score* 


unadjusted 
regression 
analysis 


Age- 
standardised 
score** 
Scale score* 


Scale score* 


adjusted 
regression 
analysis 


Age- 
standardised 
score** 
Scale score* 


Scale score* 


comprehension Comprehension 

Scale 

* The scale score is the equivalent of a raw score for an adaptive test 

** Age-standardised score has the mean standardised score of around 100 with a standard 


deviation of around 15. 


As indicated above in Table 2, descriptive analyses (for both the primary and secondary outcomes) are 
based on scale scores for pupils for whom post-test data were available. 


Hierarchical linear regression modelling was used to compute the effect sizes on both the primary and 
secondary outcomes. In models designed to test the primary outcome the age standardised scores 
were used which provide an indication of whether pupils are performing at the average level for their 
age, below or higher than average.® The scale scores (as the only available scores) were used for the 
secondary outcomes (i.e. sentence completion and passage comprehension). The modelling strategy 
used involved fitting a multi-level linear regression model with random intercepts. Pupils represented 
Level 1 and classes were indicative of Level 2. Fixed effects at Level 2 to account for the potential 
variation at school level were also used. Schools were modelled as fixed effects because the sample 
of schools was chosen pragmatically and was not selected on the basis of random sampling. The 
following covariates were included in the adjusted analysis: 


e Class level: A dummy variable indicating whether the class was an intervention class and 
a set of dummy variables indicating schools. 

e Pupil level: Pre-test scores, eligibility for FSM, EAL, ethnic group, gender and month of 
birth. These covariates that are possibly predictive of the outcome under study were 
included in the analysis to adjust for possible imbalances between intervention and 
control pupils. 


8 Age standardised scores are based on the pupils’ raw score which has been adjusted for age and placed on a 
scale that makes a comparison with a nationally representative sample of pupils of the same age across the UK. 
The average score is 100. 
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Separate pre-specified regression models were estimated for Year 7 and Year 9 pupils and for those 
who qualified for FSM. To confirm the findings with regards to FSM an interaction test was also 
implemented.° 


Following a similar approach, separate pre-specified regression models were also implemented for the 
two genders, boys versus girls (Separately for Year 7 and Year 9 pupils). The analyses followed a similar 
approach to the one described above. 


Finally, even though specified in the protocol, we unable to carry out separate analyses for pupils with 
EAL due to very low sample sizes.° Furthermore, the current report does not include SEN subgroup 
analysis on outcomes of interest. However, the project team will be carrying out further analysis to 
assess the impact of the programme on SEN pupils. 


Effect sizes and their respective 95 per cent confidence intervals were calculated following the 
procedure set out in Tymms (2004): 


fi 


es =— 
itd 


Where # represents the adjusted difference in outcomes between intervention and control groups 


obtained from the adjusted multi-level model and @ the square root of the pupil level variance obtained 
from fitting an unadjusted multi-level model. The unadjusted model contains a constant, an intervention 
dummy variable and class level fixed effect dummies, but no further covariates. The Standard Error of 
the Effect Size was computed according to Shagen and Elliot (2004). 


Finally, summary descriptive statistics are produced along with estimates of ICCs for each regression 
model estimated. 


Process evaluation methodology 


The purpose of the process evaluation was to explore and understand the experiences of schools and 
teachers in implementing the Paired Reading programme in their school. The aims were to: 


e Understand, at a strategic level, what attracted schools to take part in the programme. 

e Understand the process of joining and setting up the programme in the school including 
training, pairing the pupils and timetabling. 

e Explore how the programme worked in practice and any implementation, fidelity or 
sustainability issues. 

e Find out in which ways teachers and senior leaders perceived the programme to be 
making an impact at pupil and school level. 

e Explore how the teachers felt the programme could be developed if it were to be 
implemented more widely. 

e Look into issues around sustainability of the programme 


° The interaction models were implemented using a random intercept and random slope hierarchical regression 
model (multi-level model). We specified a random slope for the individual level variable indicating if the student 
was eligible for free school meals. This means that we allowed the effect (slope) of free school meal eligibility on 
the post-test to vary across classes. We further specified that any difference in the variance of these slopes (i.e. in 
the effect receiving free school meals has on the outcome) is due to having been in a class which was in the 
treatment group versus the control. This latter specification was included through a cross-level interaction between 
receiving free school meals and the allocation into treatment or control classes. The results of this interaction 
indicate if being in a treatment versus control class has any impact on the effect FSM has on the outcome. 

10 There are only 31 EAL pupils in Year 7 and 46 EAL pupils in Year 9 (as randomised). As analysed, the numbers 
drop to 28 in Year 7 and to 45 in Year 9. 
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A qualitative approach was used to meet these objectives and involved in-depth interviews with 
teachers and senior leaders from participating schools. 


In addition to staff interviews, the original study design included an online teacher survey to complement 
the information provided by the project team on participating schools. The results of the survey were 
intended to be used to select a representative sample of experiences to follow up in depth. However, it 
was not feasible to carry out the online survey as schools were reluctant to introduce further research 
burden in addition to already existing data collection as part of the trial. 


Process evaluation was set out to carry out interviews with two staff members from six schools including 
a mixture of both Year 7 and Year 9 teachers and senior leaders or operations managers. As indicated 
below, there were 8 interviews in total achieved as part of the process evaluation across 8 schools 
including 3 senior leaders, 3 Year 7 teachers and 2 Year 9 teachers. 


Sampling and recruitment 


The table below provides a breakdown of the achieved sample in relation to the primary sampling 
criteria. The aim was to speak to senior leaders as well as teachers. The table below provides a 
breakdown of the achieved sample by the type of interviewee. All ten secondary schools that 
participated in the programme were approached to take part in the process evaluation as they had 
previously signed a MoU with North Tyneside Council, in which they agreed to participate in research. 


Table 3: Achieved sample for the process evaluation, based on number of pupils, FSM 
eligibility and Special Education Needs 


Interview Interviewee | Number Number FSM Tod alole) | School Statement 
focus of pupils _ of pupils Action Action 
in Year7 in Year9 Plus 


Strategic | Senior 117 107 27% 10% 9% 2% 
leader 
Senior 225 184 15% 17% 9% 2% 
leader 


Senior 260 222 9% 8% 

a ae oe a 
Year 7 
pupils 
Year 9 
pupils 


Each school was contacted and provided with details of the process evaluation. They were invited to 
nominate one member of teaching staff taking part in the programme to be interviewed. An online 
booking system was set up to allow participating staff to arrange a suitable interview time. 


Individual depth interviews were conducted with five teachers and three senior leaders in eight of the 
schools. Interviews took place over the telephone and lasted approximately half an hour each. 


The eight achieved interviews covered schools with a wide range of characteristics. As the above table 
shows, schools included in the process evaluation varied in terms of their size, number of pupils eligible 
for FSM and with special educational needs (School Action, School Action Plus, Statement). This 
variation in key demographics helped give a good understanding of how the programme worked in a 
range of different school settings. 


The interviewees were also varied. Among the senior leaders, interviews were conducted with 2 deputy 
head teachers as well as a teaching and learning co-ordinator. All of the teaching staff interviewed were 
responsible for the day to day running of the programme, 
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Ideally, interviews would have been conducted with all of the schools involved in the programme. 
However, this was not possible due to the unavailability of teachers; many were too busy to take part 
in an interview towards the end of the summer term despite flexibility in times and dates offered to them. 


Carrying out the interviews 


The interviews all took place by phone. They were based around a topic guide to ensure systematic 
coverage of key issues, but were also intended to be flexible and interactive, allowing issues of 
relevance for individual respondents to be covered through detailed follow-up questioning. Follow-up 
questioning of interviewees was led by their responses, ensuring questioning was responsive and all 
issues were explored. Copies of the topic guides have been attached to this report in the appendices 
(see Appendix B). 


Two topic guides were developed: one for use with senior leaders, the other for interviews with teaching 
staff. 


The interviews with senior leaders were more focused on the school-level implementation of the 
programme: 


e Joining the peer tutoring programme - including any strategic considerations. 

e Running the programme - including cost and resources required to implement the programme 

e Response to the programme from the staff. 

e Implications of the intervention - including any perceived impacts and whether it would be 
sustained or rolled out further in the future. 


The teacher topic guide was focused more on the day to day delivery of the programme and covered: 


e Other literacy interventions in the school, including those running before or at the same time as 
the peer tutoring programme. 

e Training provided to teachers, its coverage and suitability. 

e Informing and training the pupils. 

e The process of pairing the pupils and setting up the programme. 

e The Paired Reading sessions - including the format, any barriers to successful implementation, 
progress monitoring and fidelity. 

e Perceived impact - on the pupils as well as on the wider school. 

e Proposed development, any necessary changes or recommendations. 


Analysis 


All interviews were digitally recorded and stored securely. An analysis framework was set up and the 
research team populated cells in the framework for each respondent. Following this charting, the 
framework was analysed thematically. 


Costs 


Please see page 43 for further details on the costs. 
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Impact evaluation 


Timeline 


Table 4: Project timeline 
Date Activity 


March 2013 Recruitment of schools into the study 

September 2013 RCT begins with 10 schools in North East of England 

September 2013 Reading test 1 with Year 7 and Year 9 classes at participating schools 

September 2013 Randomisation of classes into intervention and control groups at 
participating schools 

September 2013 Pairing of Year 7 and Year 9 intervention classes at participating schools 

November 2013 First teacher training session held 


January 2014 Programme delivery begins with Year 7 and Year 9 intervention classes 
at participating schools 


January 2014 Remaining classes at participating schools continue operating as usual 


Participants 


School recruitment 


Schools were recruited directly by the delivery team and were contacted by email followed by in-person 
meetings with the head teacher or senior leader. Given that the randomisation was done at class level, 
to recruit 120 classes the delivery team approached 15 secondary schools with the proportion of pupils 
eligible for free school meals (FSM) above the national average which were invited to participate in the 
trial. Of the total number of schools, ten were from within the North Tyneside local authority while five 
schools were based in the neighbouring authorities: three schools in Newcastle, one school in 
Gateshead, one school in South Tyneside and one school in Sunderland. All schools that were 
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approached met the inclusion criterion (FSM eligibility in the pupil population), however, four schools 
declined to participate. One school which had initially agreed to be a part of the research withdrew 
before the pre-test (and before randomisation) due to an imminent Ofsted inspection. 


Schools within North Tyneside were invited to participate first according to the EEF grant, followed by 
school in the neighbouring authorities with above average proportion of pupils eligible for FSM.** 


Class recruitment 


Within each school that was included in the trial, all Year 7 and Year 9 classes were eligible to participate 
in the study and thus for randomisation. However, when there were an uneven number of classes 
between year groups, the extra classes (either Year 7 or Year 9) were eliminated. (See Randomisation 
section for more details). 


All schools sent opt-out letters to parents of all children in Year 7 and Year 9 (See Appendix A). The 
letter explained to parents that their child, as part of their class, will be randomly selected to either 
receive the intervention programme and assessments, or assessments only. Parents were asked to 
inform their child’s teacher if they did not wish their child participating in the study. No parent opted that 
his/her child did not take part in the study. 


Participant flow 


Figure 1 shows the flow of participants in relevant arms through the study. There were a total of 2,736 
pupils in the study at the point of randomisation (1,370 and 1,366 in Year 7 and Year 9 respectively). 


At the onset of the trial, 58 classes in 10 schools (containing 1,313 participating pupils) were randomly 
allocated to receive the intervention and 62 classes in 10 schools (containing 1,423 participating pupils) 
to the control condition. There were 652 Year 7 pupils and 661 Year 9 pupils in the intervention condition 
and 718 Year 7 pupils and 705 Year 9 pupils in the control condition. 


After randomisation and pre-testing, no classes dropped out of the study. Thus, there was no attrition 
at the level of randomisation. However, a number of pupils did not complete the post-test. The number 
of pupils in the intervention arm that were lost to follow-up was 66 pupils (25 in Year 7 and 41 in Year 
9). The loss to follow-up in the control arm was 85 pupils (36 in Year 7 and 49 in Year 9). Overall, the 
proportion of pupils allocated to intervention and to the control group lost to follow-up was approximately 
5 per cent and 6 per cent respectively. 


In addition, we eliminated from the analysis those who did not have a valid Overall Reading score. This 
overall score is computed based on the results of the Sentence Completion and the Passage 
Comprehension scales (the secondary outcomes in this study). In the first stage of the assessment all 
pupils fill in the Sentence Completion Scale. In the second stage (based on their ability) pupils are either 
assigned to do the Passage Comprehension Scale or the Phonics Scale. However, the Phonics Scale 
does not generate scalable results and its results cannot be included in the computation of the scale 
score for the Overall Reading Scale. This is why the pupils who completed the Phonics Scale do not 
have valid results on The Overall Reading Scale and need to be eliminated. We identified 8 such 
students in the intervention groups (2 in Year 7 and 6 in Year 9) and 2 in the control groups (1 in Year 
7 and 1 in Year 9). 


The final number of pupils who had valid post-test data and were thus identified as eligible for 
unadjusted primary analysis was 1,239 (94 per cent of those allocated) in the intervention arm (625 who 


11 The 8 schools in North Tyneside also meet the above average FSM criterion. 
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started in Year 7 and 614 in Year 9) and 1,336 (94 per cent of those allocated) in the control arm (681 
who started in Year 7 and 655 in Year 9). 


However, it is important to note that further participants were excluded from adjusted primary and 
secondary analysis due to missing pupil level covariates capturing relevant socio-demographics (e.g. 
whether the pupil qualified for free school meals, EAL, ethnic group, sex and month of birth) in the data 
received from the National Pupil Database (NPD). Furthermore, pupils who did not complete the 
Passage Comprehension pre-test were also excluded. In the intervention arm, 2 Year 7 pupils and 3 
Year 9 pupils were excluded from adjusted analysis due to not having socio-demographic or valid pre- 
test data. As for the control arm, 4 Year 7 pupils and 1 Year 9 pupil were excluded from adjusted 
analysis due to not having socio-demographic data or valid pre-test data. 
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Figure 1: CONSORT diagram 


Assessed for eligibility Excluded 


Schools N=15 Declined to participate N=4 


Classes recruited 
(11 schools) 
Total classes N=130 


Excluded 
Withdrew prior to pre-test 
owing to imminent Ofsted 

inspection 

(1 school) 


Randomised Total classes N=10 


Classes Year 7 N=60 (pupils Y7 
n=1,370) 


Allocated to intervention Allocated to control 


Classes Year 7 N=29 (pupils Y7 n=652) 
Classes Year 9 N=29 (pupils Y9 n=661) 


Classes Year 7 N=31 (pupils Y7 n=718) 
Classes Year 9 N=31 (pupils Y9 n=705) 


Lost to follow-up + Invalid 
Classes N=0 
Pupils Year 7 n=25 + 2 
Pupils Year 9 n=41 +6 


Lost to follow-up + invalid 
Classes N=0 
Pupils Year 7 n=36 + 1 
Pupils Year 9 n=49 +1 


Eligible for analysis (unadjusted) Eligible for analysis (unadjusted) 


Classes Year 7 N=29 (pupils Y7 n=625) 
Classes Year 9 N=29 (pupils Y9 n=614) 


Classes Year 7 N=31 (pupils Y7 n=681) 
Classes Year 9 N=31 (pupils Y9 n=655) 


Note: A further 5 pupils in the intervention arm and 5 in the control arm were excluded from 
adjusted analyses due to missing pupil level covariates capturing relevant socio- 
demographics. 
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School characteristics 


In this section we look at the characteristics of the ten schools that were included in the trial. As 
illustrated in Table 5, most of the schools had Foundation status, with two schools being Academies 
and another two having Community School status. In terms of size, the schools included in the trial 
varied from smaller (maximum capacity of 504) to large schools (maximum capacity of 1,650). As 
indicated earlier, all schools were above the national average in terms of the proportion of pupils eligible 
for FSM. Finally, the number of pupils in each participating school who were included in the intervention 
group versus the control group is shown in Table 6. 


Table 5: School level characteristics 


Type of Highest Capacity Number Number %FSM” 
bY oy nee) FXe[=) Co) im 00] 0) | (ae) i oe) -3 
School 2 | Foundation | 18 1022, | 780 | 4202 
School3 | Academy | 16 | — 900 |_Noinfo | Noinfo | Noinfo__ 
School4 | Foundation | 18 1284 | 1015 | 5121 


School 5 | Foundation) 16 | 885 785 | 10 | 
School6 | Foundation) 18 | 1010995 512 | 15 
School 7 | Foundation) 16 | 504 460289 | 85 
School 9 | Community) 16 | 1100 690,884 | 88.1 


Table 6: Year 7 and Year 9 pupils in participating schools 


Year 7 pupils Year 9 pupils 
Year 7 FSM Year 7 Year 7 Year 9 FSM Year 9 Year 9 
total eligible | Control Treatment total eligible Control Treatment 


| by S| to7_— || SCT SCY 
ee 


“School 6 
‘School? | 64) 23 
‘School9 | 15) 45 


Class characteristics 


As discussed earlier, the randomisation into the intervention and control groups was carried out at class 
level. Table 7 illustrates the relevant differences between classes in terms of their composition. The 
average number of pupils per class (at randomisation stage) in the intervention arm is slightly lower for 
both years compared to the control arm. In Year 7, there are, on average, 22.48 pupils per class in the 
intervention group and there are 23.16 pupils in a class placed in the control group. In Year 9, the 
difference between the sizes of classes in the two arms is smaller: an intervention class has an average 
number of 22.79 pupils while a control class is slightly larger at 22.74 pupils. 


12 The pupil is either eligible for free school meals or has been eligible for free school meals at some point in the 
last 6 years or has been looked after continuously for 6 months during the year + aged 5-15 (2013/2014). 
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Table 7: Class level characteristics 


Year 7 Year 9 


Intervention Control class Intervention Control class 
class (n=29) (n=31) class (n=29) (n=31) 
652 718 661 705 


Number of pupils 
in trial 


Mean (SD) pupils 22.48 (4.80) 23.16 (5.01) 22.79 (4.60) 22.74 (4.44) 
per class 

Median (Min; Max) 23 (8; 30) 24 (13; 30) 24 (10; 29) 24 (10; 30) 
pupils per class 


FSM pupils onl Intervention Control class Intervention Control class 
claes (227) (esl ciaes (ne29) | (rea 


Number of pupils | 127 143 144 140 
in trial 


Mean (SD) FSM 4.70 (2.57) 4.61 (2.45) 4.97 (2.77) 4.52 (2.51) 
pupils per class 

Median (Min, Max) | 4 (1; 11) 4 (1; 10) 5 (G12) 4 (1; 12) 
FSM pupils per 

class 


Pupil characteristics 


The study involved the participation of both Year 7 and 9 pupils. Table 8 presents a summary of Year 
7 pupils’ characteristics in intervention and control groups as randomised and as analysed (i.e. those 
eligible for unadjusted analysis based only on the post-test scores for the primary -and secondary**- 
outcome). There were 652 pupils in intervention classes and 718 pupils in control classes at 
randomisation. Restricting analysis to just the pupils with post-test scores (as analysed) reduces the 
number. As such, there are 625 pupils in the intervention classes and 681 in the control classes. 


The baseline characteristics of pupils in intervention and control classes on the primary outcome were 
similar at randomisation and again at analysis, suggesting that the loss of pupils between randomisation 
and post-test analysis did not introduce bias on observable variables into the sample. There are slight 
differences in the distributions of scores at Key Stage 1 (at both randomisation and primary analysis) 
and there is no data for Year 7 on most Key Stage 2 English and Writing scores. 


18 Two intervention classes in Year 7 are lost when looking only at FSM pupils as in these classes there are no 
pupils who qualify for FSM. 

44 Given that the secondary outcomes are the components of the primary outcomes all students who have a score 
on the post-test of the primary outcome will automatically also have a score for the secondary outcomes. 
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Table 8: Pupil characteristics Year 7 


All pupils as randomised All pupils as analysed 
(VTL dal eLex-) co (1-) myexe) 1-9) 


Intervention Control Difference’ Intervention Control | Difference / 
(Sha (Xe 1742) (Shae 17429) 


Baseline score 99.31 99.56 -0.25 / (0.017) | 99.26 99.82 
(reading ability)*® 


Baseline score 324.20 324.38 -0.18 / (0.003) | 324.00 325.60 
(passage 

comprehension) 

Baseline score 328.90 331.40 -2.50 / (0.054) | 328.74 331.91 
(sentence 

completion) 


KS1 Reading - L1 


KS1 Reading - 24% 26% -2% 24% 25% 
L2a 
KS1 Reading - 22% 24% -2% 22% 24% 
L2b 
L2c 


KS1 Reading - 3% 3% 0% 3% 3% 
Missing 

KS2 Reading - 88% 91% -3% 88% 91% 
Expected Level 

KS2 Reading - 43% 45% -2% A2% 45% 
above a 


speaker 

Fsm 20% 20% fom met 
EverFSM 41% (40% fame ame 
SEN (20% zie te atm | 20H 
white 95% 96% 1% fume 85H 
asian 1% a fmt 
Black 4% mee 
a a a a a 


er of pupils 
(all) 


15 The difference is computed by subtracting the score in the control group from the score in the intervention groups. 
16 We eliminate those who do not have a valid Overall Reading Score due to not having taken the Passage 
Comprehension test. There are 2 such pupils in Year 7 
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Note. L = Level. There were no data available on the following characteristics: KS2 English — L2, KS2, English — L3, KS2 
English — L4, KS2 English — L5, KS2 English — Missing, KS2 English — Low attainment, KS2 English — Expected level, KS2 
English — High attainment, KS2 Writing — Expected level. 


The results of a similar exploration of Year 9 pupils’ characteristics are provided in Table 9. The 
intervention schools had 661 pupils and control schools had 705 pupils at randomisation, with these 
numbers falling to 614 and 655 respectively in the sample for unadjusted analysis. Slightly larger 
differences in average Key Stage 1 scores are observed for Year 9 pupils compared to Year 7, both at 
randomisation and at analysis. In addition, in Year 9 classes there seem to be slightly more FSM pupils 
in the intervention classes compared to the control classes both at randomisation and at analysis. The 
adjusted multivariate analysis does control for FSM. 


Education Endowment Foundation 28 


Paired Reading 


Table 9: Pupil characteristics Year 9 


All pupils as analysed 
(with post-test scores) 


All pupils as randomised 


Difference / 
(Effect size) 


Intervention (exe) ai iKe)| Difference / | Intervention (exe) ai iKe)| 
(Effect size) 


Baseline score 98.76 99.15 -0.52 / (0.035) | 99.00 100.00 -1.14/ 
(reading (0.078) 
ability)?’ 
Baseline score 348.63 349.06 -0.43 / (0.007) | 349.78 352.25 -2.47 | 
(passage (0.043) 
comprehension 
Baseline score 353.59 356.45 -2.86 / (0.058) | 354.12 358.71 -4.59 / 
(sentence (0.093) 
completion 
Li 
L2a 
L2b 
L2c 
L3 
KS1 Reading - 6% 6% 0% 5% 5% 0% 
Missing 
L2 
L3 
L4 
L5 
KS2 English - 2% 4% -2% 2% 3% -1% 
Low Attainment 
KS2 English - 84% 83% -1% 85% 85% 0% 
Expected level 
KS2 English - 26% 30% -4% 27% 32% -5% 
High attainment 
KS2 Reading - 86% 85% -1% 86% 86% 0% 
Expected Level 
KS2 Reading - 42% 47% -5% 43% 49% -6% 
Above expected 
KS2 Writing - 77% 77% 0% 78% 79% -1% 
Expected level 
‘Female | 48% ~~ 48% 47% -1% 
EAL 3% 4% 
Native English 96% 96% 0% 96% 96% 0% 
speaker 
1” We eliminate those who do not have a valid Overall Reading Score due to not having taken the Passage 
Comprehension test. There are 3 such pupils in Year 9 
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FSM 22% 
Ever FSM 


‘Mixed 

“Other 
Number of 661 
pupils (all) 


Outcomes and analysis 
Primary outcome: The overall reading score 


The analysis and results presented below first focus on the primary outcome. The two secondary 
outcomes will be discussed in subsequent sections. All outcomes analysed were pre-specified in the 
protocol. All sub-group analyses were pre-specified in the protocol as a separate FSM analysis is a 
requirement of all EEF evaluations. 


Summary statistics of the primary outcome 


Initial analysis is based only on the post-scores for the primary outcome (the NGRT Overall Reading 
Scale?®), without taking into account pre-test scores or pupil characteristics and without assessing 
whether the differences we observe in the scores are statistically significant?®. Such tests are performed 
in the subsequent sections. As the summary statistics displayed in Table 10 indicate, the mean post- 
test score on the primary outcome for Year 7 pupils in the intervention group (338.27) is very close 
(although lower) to that in the control group (341.69). As analysed, the sample of Year 7 students 
consists of just over 1,300 pupils across 60 classes. Post-test scores were missing for around 4 per 
cent of analysed pupils in intervention classes and 5 per cent of pupils in control classes. 


Table 10: Unadjusted average overall reading scores Year 7 pupils (primary outcome) 


Sample as analysed 
classes 

Sample as analysed 
pupils 


Intervention Control 


Overall Reading Scale 
Mean (SD) 338.27 (53.43) 341.69 (52.26) 340.06 (52.83) 
Median (Min, Max) 340 (144; 468) 340 (164; 496) 340 (144; 469) 


Missing (% of pupils 5.15% 4.67% 
randomised) 


The ‘as analysed’ summary statistics describing the mean and the median levels of the post-test for 
the two groups for Year 9 are displayed in Table 11. Similarly to Year 7 pupils, the difference in 


18 We excluded from the ‘as analyzed’ group those pupils who had a valid score on the Sentence Completion Scale 
but did not complete the Passage Comprehension Scale. There was 1 such pupil in the Year 7 control group and 
2 pupils in the Year 7 intervention group. In Year 9 we excluded 1 pupil in the control group and 6 pupils in the 
intervention group. 

18 For this specific analysis the scale score was used as the age-standardised score would have a fixed mean and 
standard deviation and therefore the results would not be informative. 
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average post-test scores between intervention (360.33) and control (365.87) pupils is small. 
Approximately 7 per cent of post-test scores are missing for pupils, as analysed, in both the 
intervention schools and control schools. 


Table 11: Unadjusted average overall reading scores Year 9 pupils (primary outcome) 


Talecyavcevaleceyal fexeyaline)| 


Sample as analysed n= 31 
classes 

Sample as analysed n= 655 
pupils 


Overall Reading Scale 
Mean (SD) 360.33 (57.03) 365.87 (58.17) 363.19 (57.66) 
Median (Min, Max) 368 (144; 512) 372 (168; 512) 372 (144; 512) 


Missing (% of pupils 7.09% 
randomised) 


Impact of Paired Reading on the overall reading score 


The analysis of the primary outcome was undertaken using a multi-level regression model with random 
intercepts, separately for Year 7 and 9 pupils. For each year group two analyses are presented, one 
based on an unadjusted model and the second one based on adjusted analysis. The adjusted analysis 
includes the baseline measure (pre-test) and EEF designated variables as covariates. These covariates 
are students’ month of birth, sex, eligibility for FSM and second level dummy variables representing 
schools. 


Table 12 reports estimates for Year 7 pupils. The difference between the effect of the allocation on the 
post-test score in both the adjusted and unadjusted analyses was very small and did not reach 
conventional levels of statistical significance. This indicates that there is no evidence of any impact on 
the reading competency of the tutees (Year 7 pupils) who received tutoring on reading from Year 9 
pupils as part of the Paired Reading programme The adjusted analysis was conducted on a final sample 
of 1,300 Year 7 pupils across 60 classes in all 10 schools. The adjusted analysis reveals a difference 
of -0.28 (95% Cl: -1.58 to 1.02) in the age standardised overall reading score at post-test between the 
intervention and the control group. The negative score indicates that the reading score was lower in the 
intervention group. This is equivalent to an effect size of -0.02 (95% CI: -0.15 to 0.11). 


Table 12: Analysis of the overall reading score - Year 7 pupils (primary outcome) 
Unadjusted analysis Adjusted analysis 


-0.02 
(-0.15 to 0.11) 


-0.75 -0.28 
(-4.35 to 2.85) (-1.58 to 1.02) 


0.683 

0.176 0.037 
(0.034) (0.015) 
40.74 

(9.44) 
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Variance pupil level | 190.50 
(SE) (7.65) 


Total sample size 1,305 (60) 1,300 (60) 
(classes) 


Similar analysis was performed for Year 9 pupils. Table 13 displays the results. Similarly to Year 7, the 
difference between the allocated groups on the post-test score in both the unadjusted and adjusted 
analyses was very small and did not reach conventional levels of statistical significance. Therefore, 
there was no evidence of impact on reading competence for Year 9 pupils. This means that assisting 
younger pupils in improving their reading does not make a visible difference in their level of reading 
competence. The adjusted analysis was conducted on a final sample of 1,265 Year 9 pupils across 60 
classes in all 10 schools. The adjusted analysis reveals a difference of -0.911.05 (95% Cl: -2.08 to 0.25) 
in the age standardised overall reading score at post-test between the intervention and the control 
group. The negative score indicates that the reading score was lower in the intervention group. This is 
equivalent to an effect size of -0.06 (95% Cl: -0.14 to 0.02). 


Table 13: Analysis of the overall reading score - Year 9 pupils (primary outcome) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.06 

(-0.14 to 0.02) 
Regression -1.60 -0.91 
coefficient (95% Cl) | (_4 16 to 1.00) (-2.08 to 0.25) 


ICC (SE) 0.059 0.018 
(0.018) (0.012) 
Variance class level | 14.44 1.41 
(SE) (4.71) (0.93) 
Variance pupil level | 230.81 78.58 
(SE) (9.40) (3.20) 
Total sample size 1,266 (60) 1,265 (60) 
(pupils) 


Impact of Paired Reading on the overall reading score: FSM subgroups 


In ensuring that the analysis conforms to EEF standards, a set of additional analyses were carried out. 
These subgroup analyses were performed to examine whether there was evidence that the Paired 
Reading programme had an impact on the overall reading score (primary outcome) depending on pupil 
eligibility for FSM. This analysis was carried out for both Year 7 and 9 pupils separately and was 
implemented using two distinct methodological approaches. In the first approach, both the unadjusted 
and adjusted multilevel regression models were re-estimated including only pupils eligible for FSM. In 
the second approach interaction tests were implemented to assess whether the eligibility for FSM has 
an impact on the effect the intervention (versus the control) has on the primary outcome. 


Amongst the 246 Year 7 pupils eligible for FSM there is no evidence of impact on the primary outcome 
by the allocation groups; results reveal a difference between allocated groups equivalent to an effect 
size of -0.04 but this did not reach conventional levels of statistical significance (95% Cl:-0.21 to 0.13). 
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The equivalent result for the Year 9 sample, which comprised 246 pupils eligible for FSM, was an effect 
size of -0.06and again did not reach conventional levels of statistical significance (95% Cl: -0.14 to 
0.02). The negative results indicate that the reading score was lower in the intervention group. Full 
results from these analyses are presented in Appendix D (See Tables D1a and D4a for Year 7 and Year 
9 sample respectively). 


The interaction analysis described as our second approach is also presented in Appendix D (Tables 
Dib and D4b). For both Year 7 and Year 9 the results displayed for the interaction models is fully 
consistent with the analysis discussed above and shows that the effect of allocation does not vary 
according to whether pupils are FSM eligible or not. 


Impact of Paired Reading on the overall reading score: Gender subgroups 


Along with performing subgroup analyses for pupils eligible for FSM we also carried out analysis by 
gender. To accomplish this we re-ran the analysis discussed above for each gender separately. These 
analyses were carried out to determine whether there was evidence that the Paired Reading 
programme had a different impact on the overall reading ability of boys versus girls. 


For Year 7 boys (635 pupils) there is no evidence the Paired Reading programme had an impact. The 
results reveal a difference between groups equivalent to an effect size of -0.05 but this did not reach 
conventional levels of statistical significance (95% Cl:-0.17 to 0.07). The equivalent result for the Year 
7 girls (665 pupils) was similar and also showed no impact. The effect size was equal to 0.02 and was 
not statistically significant (95% Cl: -0.08 to 0.12). The negative result indicates that the reading score 
was lower in the intervention group while the positive score shows that reading score was higher in the 
intervention group. Full results from these analyses are presented in Appendix E (See Tables E1a and 
E1b for Year 7 boys and girls respectively). These results were also confirmed by the second analytical 
strategy: the interaction analysis. The results presented in Table E1c (Appendix E) indicate that gender 
does not have an impact on the effect the allocation (i.e. treatment) has on reading ability. 


Looking at Year 9 boys (662 pupils), the results are consistent with those for Year 7 boys: the treatment 
does not seem to have had an impact on their overall reading ability. The effect size is -0.01 and is not 
statistically significant (95% Cl: -0.11 to 0.09). The same is not the case for Year 9 girls (603 pupils). 
The results show that for girls in Year 9 the treatment did have an effect on their overall reading ability. 
The effect is, however, negative, meaning that having acted as a tutor in the Paired Reading programme 
actually decreased their overall reading ability. The effect size is equivalent to -0.12 and is statistically 
significant (95% Cl: -0.22 to -0.02). The second analytical strategy comes close to confirming that, it is 
likely that the treatment had a negative effect on girls. The regression coefficient corresponding to the 
interaction term fails to attain statistical significance at the 0.05 level but it is significant at the 0.1 level 
(the actual value is 0.075). 


Secondary outcomes 


The two secondary outcomes of interest are the two components of the Overall Reading Scale. These 
are the Passage Comprehension Scale and the Sentence Completion Scale. The paragraphs below 
present the results for both these outcomes. We first assess passage comprehension and subsequently 
consider sentence completion. 


Summary statistics of secondary outcome: Passage comprehension 


Similarly to the initial analysis of primary outcome, we present summary statistics for all students with 
values on the post-test, without taking into account pre-test scores or pupil characteristics. The 
summary Statistics displayed in Table 14 indicate that the mean post-test score on the passage 
comprehension for Year 7 pupils in the intervention group (337.15) is similar to that in the control group 
(339.15). AS mentioned before, the sample of Year 7 students consists of just over 1,300 pupils across 
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60 classes. Post-test scores were missing for around 4 per cent of analysed pupils in intervention 
classes and 5 per cent of pupils in control classes. 


Table 14: Unadjusted average passage comprehension scores Year 7 pupils (secondary 
outcome) 


Talccyavceyalaceyal 


Sample as analysed 
classes 


Sample as analysed 
pupils 


339.15 (60.68) 338.19 (60.64) 
344 (148; 520) 340 (148; 520) 


5.15% 4.67% 


The results presented above are similar for Year 9 students (Table 15). In essence there does not seem 
to be any differences between the intervention group and the control group when it comes to the mean 
value of the post-test. As the figures show, the difference in average post-test scores between 
intervention (354.72) and control (361.94) pupils is very small. 


Table 15: Unadjusted average scores Year 9 pupils - passage comprehension (secondary 
outcome) 


Taceaceyaliceya fexeynline)| 


Sample as analysed 
classes 


Sample as analysed 
pupils 


361.94 (67.53) 358.45 (66.55) 
Median (Min, Max) 372 (88; 520) 372 (88; 520) 


Missing (% of pupils ‘ 6.96% 7.03% 
randomised) 


Impact of Paired Reading on the passage comprehension 


The analysis of the secondary outcomes was undertaken in an identical way to that of the primary 
outcome: a multi-level regression model with random intercepts was used separately for Year 7 and 9 
pupils. Similarly to the above, the unadjusted and adjusted models are presented. 


The findings for Year 7 pupils are shown in Table 16. Similarly to the analysis of the primary outcome, 
the difference between the effect of the intervention on the post-test score in both the adjusted and 
unadjusted analyses was very small and did not reach conventional levels of statistical significance. 
This indicates that there is no evidence of any impact on the passage comprehension of Year 7 pupils. 
The adjusted analysis was conducted on a final sample of 1,300 Year 7 pupils across 60 classes in all 
10 schools. The adjusted analysis reveals a difference of -0.48 (95% Cl: -6.07 to 5.17) in the passage 
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comprehension score at post-test between the intervention and control groups. The negative score 
indicates that the reading score was lower in the intervention group. This is equivalent to an effect size 
of -0.01 (95% Cl: -0.12 to 0.10). 


The results are similar for Year 9 students (Table 17) and indicate no significant difference between the 
intervention group and the control. The adjusted analysis was conducted on a final sample of 1,265 
Year 9 pupils across 60 classes in all 10 schools. The adjusted analysis reveals a difference of -5.22 
(95% CI: -11.95 to 1.16) in the passage comprehension score at post-test between the intervention and 
control groups. The negative score indicates that the reading score was lower in the intervention group. 
This is equivalent to an effect size of -0.09 (95% Cl: -0.20 to 0.02). 


Table 16: Analysis of passage comprehension - Year 7 pupils (secondary outcome) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.01 

(-0.12 to 0.10) 
Regression -1.60 -0.48 
coefficient (95% Cl) | (15.22 to 12.02) (-6.07 to 5.17) 


ICC (SE) 0.164 0.036 

(0.033) (0.015) 
Variance class level | 572.81 53.60 
(SE) (135.12) (22.38) 
Variance pupil level | 2929.99 1425.66 
(SE) (117.59) (57.27) 
Total sample size 1,306 (60) 1,300 (60) 
(pupils) 


Table 17: Analysis of passage comprehension - Year 9 pupils (Secondary outcome) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.09 

(-0.20 to 0.02) 
Regression -6.98 -5.22 
coefficient (95% Cl) | (17.32 to 3.36) (-11.59 to 1.16) 


ICC (SE) 0.058 0.035 
(0.018) (0.014) 
Variance class level | 230.94 66.92 
(SE) (74.96) (27.93) 
Variance pupillevel | 3724.15 1825.36 
(SE) (151.40) (74.28) 
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Total sample size 1,269 (60) 1,265 (60) 
(pupils) 


Impact of Paired Reading on the passage comprehension: FSM subgroups 


Using the same approaches presented in the subgroup (FSM) analysis of the primary outcome, we 
assessed the impact of FSM eligibility on the effect the intervention has on the relevant secondary 
outcome. 


Amongst the 246 Year 7 pupils eligible for FSM there was no evidence of impact on the primary outcome 
by the allocation groups; results reveal a difference between allocated groups equivalent to an effect 
size of -0.02 without it reaching conventional levels of statistical significance (95% Cl:-0.20 to 0.16). 
The results for Year 9 pupils (246 pupils) indicate an effect size of -0.03 without reaching conventional 
levels of statistical significance (95% Cl: -0.22 to 0.16). The negative scores indicate that the reading 
score was lower in the intervention group. As before, full results from these analyses are presented in 
Appendix D (see Tables D7a and D10a for Year 7 and Year 9 sample respectively). 


Similar results are obtained by the implementation of the second approach: the interaction analysis. 
(Tables D7b and D10b in Appendix D). For both Year 7 and Year 9 the results are consistent with the 
analysis discussed above and show that the effect of allocation does not vary according to whether 
pupils are eligible for FSM. 


Impact of Paired Reading on the passage comprehension: Gender subgroups 


The same approach (to the one used in the analysis of the primary outcome) was used to assess the 
separate impact the treatment might have had on boys’ and girls’ level of passage comprehension. 


The analysis of Year 7 boys (635 pupils) showed no evidence of impact. The effect size is equivalent 
to -0.03 and is not statistically significant (95% Cl: -0.16 to 0.10). The same can be said about Year 7 
girls (665 pupils): available evidence does not allow us to conclude that the treatment had an effect. 
The size of the effect we discover is 0.02 without being statistically significant (95% Cl: -0.10 . to 0.14). 
The positive score indicates that the reading score was higher in the intervention group. The full results 
are displayed in Appendix E (Tables E9a and E9b). The second analytical procedure (the interaction 
model) supports these findings. The results are included in Table E9c in the Appendix E. 


For Year 9 pupils the results resonate with those identified in the analysis of the primary outcome. The 
results show that for Year 9 boys (662 pupils) the treatment had no effect on their level of passage 
comprehension. The effect size is equal to 0.01 and does not attain statistical significance (95% Cl: - 
0.12 to 0.14). However, there is evidence that the treatment had a statistically significant effect on the 
level of Year 9 girls’ (603 pupils) passage comprehension (effect size: -0.20; 95% Cl: -0.21 to -0.19). 
The results indicate that for a Year 9 girl the inclusion in the Paired Reading programme as a tutor had 
a negative effect decreasing the level of passage comprehension. This finding is also confirmed by the 
interaction analysis displayed in Table E13c in Appendix E. 


Summary statistics of secondary outcome: Sentence completion 


Finally, the summary analysis of the second secondary outcome, sentence completion, produces 
results which are consistent with the findings obtained for the overall reading score (primary outcome) 
and the passage comprehension (Secondary outcome). The summary statistics indicate that the mean 
post-test score on the sentence completion for Year 7 pupils (Table 18) in the intervention group 
(340.66) is similar to that in the control group (345.41). The sample size and the proportion of pupils 
missing on the post-test are similar to the summary statistics presented for the primary outcome. 
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Table 18: Unadjusted average sentence completion scores Year 7 pupils (secondary outcome) 
Intervention 


Sample as analysed 
classes 


Sample as analysed 
pupils 


345.41 (50.11) 343.14 (49.82) 
344 (36; 504) 344 (36; 504) 


Similar results were found for Year 9 pupils (Table 19). The summary statistics indicate no differences 
between the intervention group (367.32) and the control group (371.98) when it comes to the mean 
value of the post-test. 


Table 19: Unadjusted average sentence completion scores Year 9 pupils (secondary outcome) 
Intervention Control 


Sample as analysed 
classes 


Sample as analysed 
pupils 


371.98 (55.10) 369.71 (57.23) 
Median (Min, Max) 368 (0; 536) 368 (0; 536) 


Missing (% of pupils 
randomised) 


Impact of Paired reading on sentence completion 


A similar multi-level analysis approach was implemented for the second secondary outcome (sentence 
completion) as well. Consistent with the previous analysis, once again we find no impact of the 
treatment on the post-test score of this secondary outcome. The adjusted analysis for Year 7 pupils 
(presented in Table 20) was conducted on a final sample of 1,304 Year 7 pupils across 60 classes in 
all 10 schools. The adjusted analysis reveals a difference of -1.60 (95% Cl: -6.01 to 2.80) in the 
Sentence Completion score at post-test between the intervention and control groups. The negative 
score indicates that the reading score was lower in the intervention group. This is equivalent to an effect 
size of -0.04 (95% Cl: -0.14 to 0.06). 


Table 21 displays the results for Year 9 students, indicating again, the lack of effects detectable at the 
conventional levels of statistical significance. The adjusted analysis was conducted on a final sample 
of 1,275 Year 9 pupils across 60 classes in all 10 schools. The adjusted analysis reveals a difference 
of -0.43 (95% Cl: -4.67 to 3.80) in the Sentence Completion score at post-test between the intervention 
and control groups. The negative score indicates that the reading score was lower in the intervention 
group. This is equivalent to an effect size of -0.01 (95% Cl: -0.09 to 0.08). 
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Table 20: Analysis of sentence completion - Year 7 pupils (secondary outcome) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.04 

(-0.14 to 0.06) 
Regression -4.29 -1.60 
coefficient (95% Cl) | (15.02 to 6.45) (-6.01 to 2.80) 


ICC (SE) 0.142 0.017 

(0.032) (0.013) 
Variance class level | 344.49 19.78 
(SE) (86.76) (15.21) 
Variance pupil level | 2068.95 1169.18 
(SE) (83.07) (47.12) 
Total sample size 1,309 (60) 1,304 (60) 
(pupils) 


Table 21: Analysis of sentence completion- Year 9 pupils (secondary outcome) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.01 

(-0.09 to 0.08) 
Regression -4.06 -0.43 
coefficient (95% Cl) | (11.68 to 3.57) (-4.67 to 3.80) 


ICC (SE) 0.024 0.000 
(0.013) (0.000) 
Variance class level | 77.10 0.00 
(SE) (42.10) (0.00) 
Variance pupil level | 3079.07 1447.60 
(SE) (124.97) (57.33) 
Total sample size 1,276 (60) 1,275 (60) 
(pupils) 


Impact of Paired Reading on the sentence completion: FSM subgroups 


The analysis of the FSM subgroup for the sentence completion (Secondary outcome) was carried out 
in a similar fashion to the analysis described in the FSM analysis of the primary outcome. As such, 
amongst the 248 Year 7 pupils eligible for FSM there is no evidence of impact on the primary outcome 
by the allocation groups; results reveal a difference between allocated groups equivalent to an effect 
size of -0.15 without it reaching conventional levels of statistical significance (95% Cl:-0.39 to 0.09). 
The result for Year 9 pupils (249 pupils), indicate an effect size of -0.13, which did not reach 
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conventional levels of statistical significance (95% Cl: -0.31 to 0.05). The negative scores indicate that 
the reading score was lower in the intervention group. As before, full results from these analyses are 
presented in Appendix D (See Tables D13a and D16a for Year 7 and Year 9 sample respectively). 


Finally, the results obtained using the interaction models (Tables D13b and D16b in Appendix D) 
suggest that, for both Year 7 and Year 9, being eligible for FSM does not have an impact on what effect 
the intervention has on this secondary outcome. 


Impact of Paired Reading on the sentence completion: Gender subgroups 


Similarly to previous analysis described above, separate analyses were run for each gender to test the 
effect of the programme on sentence completion (Secondary outcome). The analysis was implemented 
in a similar fashion to the previous sections. 


The results indicate that for Year 7 boys (639 pupils) the treatment seemed not to have an effect on the 
level of sentence completion. The difference between the groups that we are able to highlight is 
equivalent to an effect size of -0.03 and is not statistically significant (95% Cl:-0.18 to 0.12). The result 
for Year 7 girls (665 pupils), also show no effect. The effect size is equal to -0.03 and is not statistically 
significant (95% Cl: -0,15 to 0.12). The results (including the interaction analysis) are presented in 
Appendix E, Tables E17a, E17b and E17c. 


For Year 9 boys (669 pupils) the analysis, once more, indicate the lack of any effects. The effect size is 
equal to 0.04 and does not attain statistical significance (95% Cl: -0.07 to 0.15). Looking at Year 9 girls 
(606 pupils) we also find no effects. It appears that when it comes to sentence completion the treatment 
does not have an effect on Year 9 girls who act as tutors. The effect size is equal to 0.06 but does not 
attain statistical significance (95% Cl: -0.18 to 0.06). The interaction model supports this finding. The 
results are included in Appendix E, Tables E21a, E21b and21c. 


Cost 


The costs connected with the delivery of the Paired Reading programme have been provided by the 
delivery team. Several assumptions have been made when considering the cost of the programme to 
schools. These include: 


e Delivery to a medium sized school with 5 class entry per year group. 

e All Year 7 and 9 pupils would take part with an average of 25 pupils per class, resulting in 250 
participating pupils per school. 

e The programme involves 30 minute sessions once a week and lasts 16 weeks. 


Training and support costs. Professional development and additional support for staff is 
recommended, particularly in the early stages of setting up a programme. It is estimated that this would 
cost a total of £2,000 per school. 


Materials required for schools would be a copy of the manual for each teacher involved and resources 
for pupils to use (including question and praise cards). The costs for these would be around £625 per 
school (or £5 per pupil pair, £2.50 per child). 


There was no additional workload intended for teachers to deliver the programme so no extra costs 
were associated with this. 


Based on the assumptions listed above the cost of the programme delivered over 16 weeks, is £2,625 
per school. This translates into a unit cost of £10.50 per pupil per academic year. 
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Process evaluation 


The process evaluation helps us to address three important questions: 
1. Were there issues in setting-up the Paired Reading programme? 


This question helped us to understand the process of setting up the programme in the school; 
including the strategic decision to join, the pairing of pupils and finding space in the timetable. 
It also helped us to examine how the teachers who delivered the programme in the schools felt 
these aspects could be improved if it were to be implemented more widely. 


2. Was there any variation in fidelity to the running of the Paired Reading programme? 


This question helped us to explore how the programme and ‘Paired Reading’ classes worked 
in practice and any implementation, fidelity or sustainability issues that were faced by the 
schools involved. It also helped us to look into issues around the sustainability of the 
programme. 


3. Were there any perceived benefits reported by staff to pupils' reading ability, or any wider 
impacts of the programme? 


This question helped us to find out how teachers and senior leaders in schools perceived the 
programme to be making (or not making) an impact; whether on the pupils or on the school at 
a wider level. 


Implementation 


This section covers issues around preparing to deliver the programme in schools, including separate 
sections about: 


e Why schools took part in the Paired Reading programme 
e Literacy interventions already in place 

e Teacher training 

e Training pupils and Introducing the programme to them 
e Pairing of pupils 

e Timetabling the programme sessions 


Participating in the Paired Reading Programme 


Schools were initially approached by North Tyneside Council to take part in the Paired Reading 
programme. The decision to participate was guided by either pupil-based or school-based factors. 


Pupil based factors: 


e Reading levels. The programme was seen as an opportunity to improve reading levels of 
both year groups. Senior leaders hoped that Year 7 pupils’ reading would improve by 
being mentored by an older pupil while the Year 9 pupils would also improve due to the 
process of mentoring and correcting a younger pupil. 

e Confidence. Senior leaders hoped that the programme would be a pleasurable process 
that would build the confidence of pupils. They hoped it would help to build relationships 
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between pupils across different age groups and promote a friendly community 
atmosphere around the school. 


School based factors: 


e Staff development. The programme was seen as an opportunity for the teaching staff to 
develop professionally. Senior leaders wanted the teaching staff to lead on a project and 
develop their own leadership potential. 

e Research participation. Schools were interested in taking part in research and trialling a 
promising approach which had previously been shown to have positive impact on student 
performance. 


“We’re a teaching school. Part of our drive is looking at things that actually work. What is behind 
teaching and learning is really important to me” 
(Senior leader) 


These were also some apprehensions about taking part: 


e Administration and extra work - There was a perceived risk that the programme may increase 
the administrative burden for schools. 

e Timetabling - Some schools felt they would have difficulty fitting the programme into their 
timetable. 


Teachers and senior leaders were largely positive about taking part in the programme. It was seen as 
an opportunity to do something different, though for some the idea of the peer mentoring programme 
was So radically new and unprecedented, they could not imagine how it was going to look, or be run, in 
practice. 


Other literacy and reading interventions 


It is important to consider that Paired Reading was not the only literacy intervention that was taking 
place in the participating schools. Schools already used a variety of different techniques and 
interventions with a view to improving literacy with their pupils. 


The existing interventions in the schools (‘business as usual’) included: 


e Literacy Clubs: These tended to occur weekly and were available to all pupils. The pupils would 
come to the club and read a book together with the others, taking turns reading passages from 
the book. They would then, as a group, discuss the book with a member of staff. 

e Additional reading support groups: These were run by either staff within the school or volunteers 
from the community and involved one-to-one literacy support to pupils who had been identified 
as requiring it. They would take place outside of school hours. 

e Withdrawal groups: These were run with pupils who were identified as requiring additional 
support with their reading. The pupils would be withdrawn from a standard timetabled class 
(usually a foreign language class) and given intensive literacy support by support assistants at 
the school. 

e Specific ‘Literacy’ lessons: Schools also included ‘Literacy’ as a standard lesson in the timetable 
where the class would read excerpts from novels and cover other skills such as spelling, 
punctuation and paragraphing. 

e Dedicated reading time: Reading was considered to be a very important part of the syllabus 
with some schools dedicating 15 minutes of every English lesson purely to reading, with an 
‘accelerated reader’ hour on top. Schools that followed this were having at least two hours of 
reading per week with their pupils. The schools involved also gave significant importance to 
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literacy as part of their syllabus. This suggests that the schools involved had a particular interest 
in literacy, which may have influenced their decision to join the pilot. 


Introducing the programme to the pupils 


Teachers were responsible for introducing the Paired Reading programme to their pupils. They 
described presenting the programme in a way which would appeal to their pupils to encourage 
enthusiasm from the outset. 


Some of the schools chose to inform their pupils by organising a special assembly (for all Year 7s or 
Year 9s separately, or both together) whereas others introduced it to individual classes. All schools 
focused on the individual benefits the programme would potentially bring pupils and particularly their 
reading ability. 


Some schools felt that this programme tied in well with the Citizenship syllabus, where the pupils were 
taught personal skills, communication and being a responsible citizen. The school linked the Paired 
Reading programme with the Citizenship syllabus and explained to the Year 9 pupils that it was part of 
their role as citizens in the community. 


‘They were helping others and benefitting themselves in the process’ 
(Teacher) 


Teachers described a mixed reaction from the pupils once they had been informed of the programme, 
although generally they were positive towards it. 


‘| thought they were really good. The Year 9s liked the responsibility they had been given over the Year 
7s and the Year 7s enjoyed having that reading buddy to talk to and do a bit of work with’ 
(Teacher) 


There were some difficulties especially with Year 9 pupils not understanding how they would benefit 
from the programme. Running the programme outside of normal timetabled hours or replacing private 
reading time also caused resentment towards the programme in some cases. 


Training the pupils 


Schools were advised to train both tutors and tutees together to ensure that they all received the same 
message and that the idea of ‘we are all learning together’ was immediately conveyed. It was 
recommended that the training time be specified well in advance, so pupils could look forward to the 
experience. As well as a training session, schools were also advised to use the first four sessions of 
the intervention as a gradual introduction. 


In practice, some schools used the resources provided during the teacher training, in order to train the 
pupils (generally feeling that the resources, whilst aimed at adults, were appropriate for Year 7 or Year 
9 pupils to use). However, while the pupils could ‘understand’ the resources, there was a view that 
pupils did not find the materials particularly exciting. Other schools therefore created their own 
resources. Many teachers also used the online videos provided by the delivery team in their training. 


On the whole, schools did not report any major problems with training the pupils. It was suggested that 
the delivery team could produce materials that were more engaging for the younger age group which 
would help to deliver more consistent messages to the participants and excite them about the 
programme. 


Pupil training took place either with Year 9s and Year 7s together, or the two age groups separately. 
The delivery team suggested that year groups of pupils are gradually introduced to the programme 
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together, step by step but there was further variation in terms of the way schools trained pupils which 
are outlined below. 


Gradual introduction. The training took place as the intervention started (as prescribed) but its length 
varied from school to school. Some schools used just the first lesson to train the pupils, while others 
provided a staged introduction to the programme, with the first few lessons dedicated to training and 
practice activities. While this may have reduced the dose of the intervention for the pupils, teachers felt 
it was necessary for pupils to fully get to grips with the programme and what was required of them 
before embarking on it. 


‘The first lesson we got them reading together, the second lesson we introduced questioning and the 
third week we did a trial and the programme actually started in the fourth week’ 
(Teacher) 


Only training Year 9 pupils. Schools spent the first two lessons or so going through the PowerPoint 
presentation with Year 9 pupils, explaining how the programme should be run, the rationale behind it, 
the structure and what their approach should be. They were told how they should support the Year 7s, 
give verbal feedback and use the question cards. Schools that took this approach then needed a lesson 
to practice before embarking on the programme fully. The Year 7s did not receive any additional 
training. 


No advance pupil training. One school took the approach of not training the pupils at all. In this case, 
schools spent a little bit of time in the first lesson going through the five finger test and selecting the 
books. However, after this, pupils went straight into the programme and were considered ‘ready to go’. 
Though this was not by any means the prevailing approach taken by schools, it was felt in this case 
that the pupils did not require an extended training or preparation period. 


Pairing pupils 


Prior to starting the programme all pupils in Year 7 and Year 9 took a literacy test to establish their 
reading ability. For the purposes of the project this assessment was also the pre-intervention test to 
avoid additional testing. Year 7 and Year 9 classes were paired based on these test results by the 
evaluation team. 


Teachers were responsible for pairing the pupils individually. The delivery team suggested that reading 
test results should be the primary factor considered when pairing pupils. They also suggested that 
teacher knowledge (such as the maturity and behaviour of the pupil) and outside factors including 
attendance should be taken into account. In practice, teachers used a mix of test results, teacher 
knowledge and other outside factors when pairing the pupils individually: 


e Test results only: Pupils with the closest reading ages were paired together. No other factors 
were taken into account. 

e Test results and teacher knowledge: Test results and the personalities of the pupils and whether 
they were likely to get along with each other were considered. 

e Test results, teacher knowledge and other outside factors: Factors based on individual (non- 
academic) circumstances which would make it inappropriate to match pupils together such as 
a dispute between the families of the pupils, were considered along with test results and other 
teacher knowledge. 


When schools had an odd number of pupils in either Year 7 or Year 9 they the delivery team suggested 
to put some in trios rather than pairs. This was also the case when one of a pair was absent. A trio 
could be with two Year 7s with a Year 9 pupil or two Year 9s with a Year 7 pupil. In practice, teachers 
used this method and in one case, when a Year 9 pupil was absent a teaching assistant would take 
their role. 
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Timetabling the programme 


There was no guidance on which particular lessons, within the school day, the intervention should 
replace. Schools were found to have timetabled the programme in the following ways: 


e Replaced an existing class - Schools took this approach out of choice or necessity: 


o Some had an appropriate class that could be replaced, such as an English lesson or 
Citizenship lesson, which they felt worked well with the programme. 

o Others replaced a modern language and ICT class. It was felt that the reading levels 
of the pupils were more important than progress in these subjects and they did not have 
sufficient registration time or timetable flexibility to make alternative changes. 


‘By splitting the lesson in half it created a very staggered lesson. If we’re doing Shakespeare, you go 
through the scene, you’re about to start the work and then it’s paired reading, so it will have to be 
pushed back to the next lesson’ 

(Teacher) 

e Replaced registration or tutorial time - This was an easier task for schools that had a specific 
time set aside in the mornings for this purpose. If schools had 20 minutes or more tutorial time 
available they fitted the programme into this period. 

e Ran the programme outside of school hours - Where not possible to replace an existing class 
or run the programme during registration or tutorial time the only remaining option was seen to 
be running the programme outside of the normal timetable hours. One school (6 classes) 
brought participating pupils in early one morning a week and held the programme before other 
lessons began for 15 minutes pre registrations and 15 minutes during registration. This relied 
on the goodwill of the teaching staff to do so. 


Staff interviews indicated that timetabling was felt to be easier for schools that were aware of the 
programme requirements before they set the timetable for the period covered. It was not always easy 
to replace an existing class in a set timetable. Some schools reported disruption when the programme 
replaced an existing class, or part of a class. There was also concern about pupils missing other 
lessons. When the intervention group was doing Paired Reading for half a lesson once a week, these 
pupils missed a lot of other lesson time over the 16 weeks of the programme. It also required teachers 
to plan their lessons in a different way to allow for certain pupils missing sections of teaching. 


This inconsistent timetabling of the programme amongst schools meant that the length of sessions, and 
the amount of time each pupil spent reading, varied between schools. Some schools were able to fit in 
a 40 minute Paired Reading session while others had to add a 20-minute slot onto the school day. This 
meant that pupils from different schools were receiving varying amounts of Paired Reading. 


Wider programme constraints 


The main issue that was raised by members of the senior leadership teams concerned the levels of 
administration the programme entailed. This related mainly to the evaluation of the programme, rather 
than the programme itself. While senior leaders did understand that the programme was being trialled 
with a view to being extended elsewhere, it appeared there was a misunderstanding about which 
aspects of the administrative tasks related to the programme itself and which aspects related to the 
evaluation of the programme. 


Senior leaders at schools listed the following as some of the administrative tasks their staff have had to 
complete during the programme: 


e Case studies 
e Logs/diaries 
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e Questionnaires 
e Reading tests for all pupils 


There were occasions when the amount of administrative tasks had an effect on the school. For 
example, one senior leader was forced to give the teachers involved with the programme a free period 
to allow them to catch up with and get through the paperwork. The senior leader felt that, without this 
free period, there simply would have been too much for them to do. There was a view that the many 
different strands of data collection and administration that needed to be coordinated were putting a 
strain on the staff involved. 


‘Now and again | felt there was a lot and the staff were under pressure. At times | felt there was possibly 
too much being put on us’ 
(Senior leader) 


The administrative aspect of the programme and its evaluation led to the following constraints being 
raised: 


e Programme length - There was a view that the programme went on too long and this resulted 
in an erosion of the enthusiasm that the staff and pupils initially felt. 


‘By the end, | think everybody was getting a little bit sick of it, the students were getting bored and for 
the staff the novelty wore off, it was just a little bit too long’ 
(Teacher) 


e Busy time of year - Teachers commented that the programme was held at a very busy time of 
the year. The second and third terms are when schools are preparing Year 9 pupils for their 
GCSEs, which will begin in the next academic year. There was a view that the Paired Reading 
programme added to teachers already large workload and made it difficult for them to fully 
engage and monitor the progress of the pupils formally. 

e No instruction for assessment - Teachers reported receiving no instruction on how to assess 
the pupils as the programme progressed, either formally or informally. This was particularly an 
issue where schools had half of their form group with another teacher for their programme 
sessions and so were not aware what progress was being made. 


Fidelity 


This section looks at issues around the delivery of the programme, including levels of fidelity and any 
barriers that schools faced with delivery. 


e Book selection 

e Lesson content and format 
e Role of the teacher 

e Programme issues 


Book selection 
Participating schools took two different approaches to select books to read: 


e The Year 7 pupils would select a book from among the school’s own resources. The pupils 
would then use the ‘Five Finger Test’ (See Intervention section above) to ensure the book was 
neither going to be too easy nor too difficult for them to read. 

e The teacher would pick the books for the pupils by suggesting two or three books to each pupil 
and then allowing the pupil to choose which one they would like to read. 
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The reasons schools chose to select books for pupils, instead of relying on the ‘Five Finger Test’ were 
around not feeling that this technique helped to find a suitable book. 


‘Occasionally they would do the five finger test and still find a book that is too hard’ 
(Teacher) 


It was also apparent that teachers’ understanding of how to do the ‘Five Finger Test’ test varied. 


‘ 


‘Any reader could pick up a book, put their five fingers across and find ‘and’, ‘or’, ‘because’, ‘but’, ‘other’ 
and think ‘Oh I can read this book’. So at secondary level there needs to be a far more rigorous way. 
There need to be recommended novels. It doesn't work at all’ 

(Teacher) 


The programme resources did not state that pupils had to choose a book from the school. On the 
contrary, they suggested pupils had a choice of material, including for example, football match 
programmes. However, it was not always possible for pupils to bring their own books or reading 
materials from home. Teachers commented that many of the pupils from deprived backgrounds would 
not have access to books at home. For many of them, school would be the only time that they had 
access to books and the opportunity to read them. 


The programme guidance was for the younger pupil to select the book. However, some schools tried 
to allow the Year 9 pupils to choose the book. The idea was that this would help the older pupil to 
become more engaged as they would be more interested in the reading material. More often than not, 
however, the book chosen was not suitable for the Year 7 pupil to read. 


Teachers felt that the best solution for the selection of books was for Year 7 pupils to choose the book 
from a recommended reading list based on reading age and covering a variety of genres. 


Lesson content and format 


Overall, teachers felt that the Paired Reading sessions worked reasonably well. This was despite 
apprehensions about having a large number of pupils reading out loud in the same room. 


At first | thought it would be far too much noise to have all those pupils reading but it was actually fine’ 
(Teacher) 


Teachers felt the programme took a little time for the pupils to get used to and for them to use the 
prescribed techniques. 


‘It worked for the most part really well but it took a long time to engrain. For some groups we felt like we 
were just getting it and the programme came to an end’ 
(Teacher) 


Whilst teachers found that some pupils were very good at following the steps, having a reminder in front 
of them certainly helped. Teachers felt that without a step-by-step guide pupils (especially those with 
lower ability levels) would have struggled to remember what to do. Teachers were also breaking the 
sessions into several shorter slots of reading, to give lower ability students more frequent breaks in the 
task. 


Correcting pronunciation. Tutors were good at correcting the pronunciation of words. The Year 9 
pupils were correcting the pronunciation of words but were not explaining the meaning to the tutee, nor 
providing an example of its use in another sentence. Teachers felt this was a missed opportunity to 
develop both pupils’ literacy. 
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Some pupils found the correcting process more difficult. There were examples of the tutors waiting until 
the tutee said they were stuck before correcting it. These pupils needed extra encouragement to read 
together with the Year 7 pupil and follow the steps correctly. 


Questioning. There were several sets of question cards that pupils could use depending on ability 
level. As aresult of this, higher ability pupils were asked more difficult questions than lower ability pupils. 
Examples of the suggested questions include: 


e Which of the characters is most like you, and why? 
e How would you cope if you were transported into the story? 
e Describe what might happen. Can you justify why you think this? 


Schools took two different approaches to this aspect of the programme: 


e There was a set time in every class when pupils would be instructed to stop reading. The Year 
9 pupils would then spend 5 minutes or so asking questions to the Year 7 pupils. 

e There was no set time with Year 9 pupils being instructed to ask questions at the end of a 
chapter or at the end of an appropriate page. 


The success of the questioning aspect of the lesson was felt to depend on two factors: 


e Lesson length - The longer the lesson that the programme replaced, the more successful this 
aspect was. Schools with shorter sessions carried out less questioning and sometimes included 
questioning at the start of a session about what pupils had read in their previous lesson. This 
was seen by teachers to make this aspect of the programme more difficult for the tutees, as the 
story would not have been as fresh in their mind as if they had just been reading the book. 


‘The writing part was absolutely fine, they were articulating it but we only had 20 minutes so that was 
one of the parts that was cut short because we needed more reading time. With an extra five minutes 
that would have been a lot more effective’ 

(Teacher) 


e Question cards - Without these, teachers felt that most pupils would have struggled to think of 
or ask appropriate questions. The majority of pupils only used the question cards and did not 
use their own questions. The success of this aspect of the lesson was therefore felt by teachers 
to depend on how well the tutor used the question cards and asked the appropriate question at 
the appropriate time. Higher ability pupils were more likely to move away from the question 
cards and think of their own questions to ask, which the lower ability pupils were unable to do. 


Praising. The Year 9 pupils had ‘praise cards’. These were used to praise the Year 7 pupils when they 
read well or pronounced a difficult word correctly. Teachers at participating schools felt that the Year 9 
girls were better than the boys at this, as the boys tended to struggle to go beyond saying ‘well done’. 
The boys in Year 9 needed more encouragement from the teacher in order to praise effectively. 


Log books. Pupils and teachers had to fill in a log book at the end of each lesson. These recorded the 
book and amount of reading done, what pupils had read well and what they needed to work on for the 
next lesson. Examples included: 


‘Work on your pace’, ‘Try to de-code words better’, ‘Pronounce your Ts’. 


Teachers reported that the students with lower literacy abilities struggled with this part of the 
programme. They tended to write the same thing in their log books every week. 


‘They didn’t have the literacy levels to vary what they wrote after each lesson’ 
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(Teacher) 


Teachers felt there was a need for more differentiation between higher and lower ability pupils. This 
could be achieved by providing less scaffolding to the higher ability pupils but more for the lower ability 
pupils to better support their learning. The schools overcame this by the teacher or teaching assistant 
providing extra help to pupils who needed it. 


Senior leaders and teachers also felt that some of the pupils did not understand the need to fill in log 
books at the end of each session. This was something that required behaviour management and 
persuasion from the teacher to ensure it was done properly. 


Pupil pairings and the use of trios. Overall, teachers felt trios, when they had to be used, worked 
well, however they identified some problems which were not present with pairs: 


e When two Year 7s were together with one Year 9, it slowed down the amount of reading 
covered, as the Year 7s were each taking turns to read their book. 

e The Year 7 pupils tended to get bored when they were not reading. This resulted in behaviour 
issues with pupils talking to or distracting others. 

e It was difficult for one Year 9 pupil to focus on the reading of both Year 7s. Teachers felt this 
resulted in less correction and the quality of the ‘buddying’ being lowered. 

e It was also difficult when there were two Year 9s with a Year 7. Not only was it intimidating for 
the Year 7 but one of the older tutors would invariably become more dominant and the other 
would become bored and distract others. 


‘They had to sit squashed on either side of this poor Year 7 and some of our year 9s are huge and the 
Year 7s are tiny so it was a little intimidating for them’ 
(Teacher) 


Role of the teacher 


The teacher had an important role in delivering the programme. This included demonstrating the 
programme approach and monitoring pupil progress. The monitoring of pupil progress occurred by 
observing and listening to them as the programme progressed but also by reading the pupils’ log books. 
Teachers were instructed to intervene during their monitoring of pupils but only when a pair was having 
difficulty to the point of being unable to move forward with the process. However, there were times when 
teachers needed to take a more pro-active role with certain pairs. This may have been due to personality 
clashes (such as over-dominant partners or gender issues) or poor communication skills. Furthermore, 
teachers were also instructed to intervene if pupils had selected books that were either easy or too 
difficult. The manual suggested that teachers take turns at observing pairs with each observation lasting 
approximately six minutes. 


One of the issues teachers faced when monitoring the progress of the pupils was that of a lack of time. 
This was a particular problem for those schools which had shorter Paired Reading lessons. 


As for programme related issues, teachers felt there were some issues which limited how closely the 
schools were able to adhere to the prescribed approach. 


Teachers additionally identified a number of perceived barriers to pupils’ tutoring or being tutored: 


e Cross-age tutoring - There were some concerns about the social aspect of the programme, with 
pupils of different ages finding it difficult to work together. This was caused by either or both of 
the year groups feeling shy or uncomfortable with the other. Some Year 7s had not previously 
interacted with Year 9 pupils in any way before the start of the programme. 
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‘Our students aren’t on vertical tutor systems. It is very horizontal and the Year 9s and Year 7s very 
rarely mix together’ 
(Senior leader) 


For many pupils, the first Paired Reading lesson was the first time they had interacted or worked with a 
pupil from a different year group. It was felt schools may need a little more time for the groups to get to 
know each other so that the social element was less of a constraint in the running of the programme. 


e Personality clashes - Some of the pupils simply did not get along very well, or were disruptive 
when paired together. Teachers found that this affected the entire class and so they tried to 
pre-empt or deal with issues as soon as possible. 


‘If it had been students who had volunteered or with Year 9s and Year 11s, it may have been better’ 
(Teacher) 


e Adhering to the approach 


The programme was felt to contain a lot of information for the Year 9 pupils to take in, particularly when 
they first started. There was a view that immaturity was preventing some Year 9s from engaging with 
the programme. It was commented that they were not mature enough to understand that they were in 
a role of leadership and responsibility. To counteract this, some of the schools held separate training 
sessions for the Year 9 pupils. However, it was felt that a more formalised training process for the Year 
9 pupils would help them to understand their role more clearly. 


e Being paired with lower ability Year 7s 


When lower ability Year 9s were paired with lower ability Year 7s teachers felt neither improved their 
reading ability nor their confidence. It was often the case that the lower ability Year 7 had a higher 
reading age than the Year 9 and this resulted in a considerable loss of confidence and morale for the 
Year 9 pupil. Lower ability pairings which were struggling with their reading, also tended to have 
behavioural issues during the class. 


‘From that they thought they’re rubbish and they're crap. Why should they bother? Why should they 
have to do it if they're not going to get any better?’ 
(Teacher) 


Perceived impacts 


Teachers and senior leaders identified a number of areas where they felt the programme had an impact 
on the school and the pupils, both positively and negatively. 


Literacy and transferable skills 


e Literacy. Although the impact evaluation has found no significant impact on the reading levels 
of participating pupils, teacher’s commented on the ease at which some of the Year 7 pupils 
were reading at the end of the programme. Some of the pupils also finished the books they 
were reading previously and were moving onto, what the teachers described as, ‘harder books’. 
Not all senior leaders we spoke to were convinced that the programme had, or would, improve 
literacy levels among the pupils due to the lack of directed teaching. 


‘Two kids reading together is fine but you can’t get a Year 9 kid to teach a Year 7 kid how to decode a 
word’ 
(Senior leader) 
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e Transferable skills. Teachers and senior leaders spoke of the transferable techniques that 
had been gained from this programme and used in the broader curriculum. For example, the 
questioning structure which teachers believed made the pupils think more about their reading. 


Non-cognitive skills 


e Improved relationships. The programme was felt to improve relationships between pupils with 
the pairs having a positive social experience from peer tutoring. In some cases, teachers felt 
that pupils who had never spoken to someone two years above them had gained in confidence 
as a result of their interactions on the programme. 

e Confidence of pupils. Teachers spoke of a visible improvement in the confidence of some 
pupils, in both years, who took part in the programme. The prevailing view was that, whilst at 
the beginning some of the Year 7s may have felt a little intimidated, by the end the fluency with 
which many of the pupils were reading, questioning and praising, had improved. 

e Loss of confidence for lower ability Year 9s. While some pupils in Year 9 gained in 
confidence, teachers felt that especially for the lower ability Year 9 pupils, the programme had 
the opposite effect. It was felt they were embarrassed about having a reading ability the same 
as or lower than a Year 7 pupil and so disengaged from learning. 


Wider school impacts 


e Up-skilling teachers. Senior leaders felt that the programme had also brought the added 
benefit of up-skilling teachers in the literacy element of teaching. This fell into the wider school 
improvement plans for schools. 


‘You look at literacy for school and people think it sits with the English department but we’ve had a big 
push on that. Everybody is a teacher of literacy’ 
(Senior leader) 
e Profile of teachers. Senior leaders felt that the programme had provided a clear focus to the 
teachers and those who ran the programme now have a higher profile within the school. 


Pupil attainment 


e Students falling behind. Teachers felt that participating pupils fell behind their peers in 
subjects that the Paired reading sessions replaced. In total, they were taken out for at least 30 
minutes a week over 16 weeks and there was a worry that this may result in them doing worse 
in their assessments than the control group pupils. 

e Impact of existing programmes. Some senior leaders felt that their school’s own existing 
literacy programmes were better than the Paired Reading programme. 


Formative findings 


There were mixed reports on whether participating schools would continue with the Paired Reading 
programme. In general schools wanted to see whether it had an impact on the reading ability of students 
before they committed to continuing with the approach. 


Schools were concerned about the implementation of the programme including timetabling the lessons. 
There were concerns that missing other subject lessons was having an adverse effect on attainment. 
This would clearly influence schools in whether they removed pupils from other lessons, or held 
sessions during tutorial time, or added sessions on to the school day. 
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However, the concept of peer tutoring was very well received and schools were more likely to continue 
with this practice. It was the fully prescribed approach of the Paired Reading programme which schools 
found challenging to follow when implementing the approach routinely in their school. 


Members of the school senior leadership teams and teachers generally felt that the concept of the 
Paired Reading programme was sound. However, for the programme to be successful, the following 
aspects should be considered: 


e Programme length: Some teachers and senior leaders felt that the programme needed to be 
shortened to around a term’s length, rather than 16 weeks. It was argued that, for the pupils, 
Paired Reading felt like a regular lesson but with 16 weeks of repeating the same thing. Some 
pupils therefore lost focus and enthusiasm. This was further compounded by the rigidity of the 
method, with little scope for variation. 

e Timetable constraints: The programme would easily slide into a school that has a significant 
time for registration in the morning. However, schools that do not have a ‘collapsed’ lesson 
struggled to fit itin and would need to see evidence of impact before committing to a change in 
the way they set up the timetable. 


The programme was perceived to have the potential to improve the reading levels of pupils at a 
minimum cost to the school. Teachers commented that the gold standard in terms of a child’s ability to 
succeed in life is their ability to read. Teachers and senior leaders identified a number of changes which 
they felt the programme should adopt if it were to be rolled out further in future: 


e Motivation for the Year 9 pupils to take part in this programme. One teacher suggested that 
they be given a certificate at the end of the programme to commend their participation. 

e Year 9 pupils need some additional support in the form of mentoring training to help them 
understand and develop their tutoring role and reduce the burden on them as they embark on 
the programme. 

e There was a suggestion to use ‘colour coding’ for the question cards to help pupils differentiate 
their difficulty more easily. 

e There needs to be more scope to develop ‘understanding’ of words, rather than simply how to 
pronounce it. Teachers commented that the best way to learn how to pronounce a word is to 
know what it means and how it can be used in different circumstances. This element was seen 
as being necessary to include in a literacy programme for this age pupils. 


Control group activity 


While pupils in intervention classes were taking part in the Paired Reading programme, the pupils in 
remaining Years 7 and 9 classes continued their ‘business as usual’. This meant that they continued in 
their normal timetabled lessons and normal classes. The pupils in control group classes may have also 
been receiving additional literacy support, such as those discussed in section ‘Other literacy and reading 
interventions’ section of the process evaluation chapter. Some of the schools were also running other 
targeted literacy interventions for pupils, which intervention and/or control group children could have 
been receiving. These data were not collected. 
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Conclusion 


Key Conclusions 


This evaluation does not provide any evidence that the Paired Reading programme had an 
impact on overall reading ability, sentence completion and passage comprehension of 
participating pupils. 

There was no evidence of the Paired Reading programme having an effect on overall reading 


ability, sentence completion and passage comprehension of FSM pupils. 

There was some variation in the intervention group schools in terms of the programme set- 
up and delivery. There was also a varying level of support provided to pupils within the 
intervention by the teachers involved, based mainly on the reading ability of the pupils. 
However, these appear to be natural variations between the settings of the schools involved 
and are unlikely to have affected the dosage of the intervention for the pupils involved. 


This evaluation has sought to provide new evidence on the effectiveness of an intervention that uses 
cross-age peer tutoring method to improve the reading ability of pupils in Year 7 and Year 9. The results 
of analysis suggest that the Paired Reading programme had no positive effect on the reading ability of 
pupils participating in the programme and a negative effect on Year 9 girls’ outcomes. 


Limitations 


Considering that all pupils in intervention cohorts were intended to receive the Paired Reading 
programme, the pupil recruitment was successful. There was no attrition at the level of randomisation 
(i.e. classes). As for pupils, 95 per cent of those allocated to the intervention group and 94 per cent of 
those allocated to the control group were eligible for unadjusted primary analysis yielding a pupil level 
attrition rate of 5 and 6 per cent respectively. Moreover, the sample characteristics in terms of 
observable variables were similar for the randomised and the analysed sample and any differences in 
intervention and control group composition are likely to be due to chance. 


It has to be acknowledged that the key limitations of this cluster randomised trial design is that 
randomisation at class rather than school level increases the possibility of ‘contamination’ (exchange 
of information among pupils and teachers in different trial groups) and increases likelihood of 
collaboration among teachers within each school, and thus the chances of treatment diffusion (i.e. some 
aspects of the experimental stimulus such as the intervention are passed on from the intervention group 
to the control group). 


This trial was undertaken within a narrow geographical area of the UK. More specifically, the trial was 
run in three LA areas in North East England — North Tyneside, South Tyneside and Sunderland. The 
Paired Reading programme was delivered by teachers in a regular school context which makes the trial 
results more applicable to a real-world setting. The proportion of FSM pupils in trial schools ranged from 
approximately 10 per cent to 35 per cent. On average 21 per cent of pupils in trial schools were eligible 
for FSM which is higher than the national average in state funded secondary schools (15 per cent, DfE, 
2014b). As for pupils with SEN, the trials schools had a higher percentage of pupils in this group 21 per 
cent), compared to 18 per cent across all pupils in state funded secondary schools (DfE, 2014c). 
Furthermore, the trial schools had much lower percentage of pupils with EAL (less than 4 per cent) 
compared to the national average (14 per cent, DfE, 2014b). Overall, the sample characteristics 
suggests that the schools participating in the study were slightly more disadvantaged than the average 
secondary school in England (as intended) while having less EAL pupils when compared to the national 
average. It also has to be acknowledged that the pragmatic approach taken to the trial design is likely 
to mean that the treatment effect is estimated on more engaged schools (and classes within schools) 
where the logistics of the standard school day could have affected how the classes were paired to allow 
easier implementation of peer tutoring. 
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It has to be noted that as part of the process evaluation, a short online teacher survey was originally 
planned to gather further information from teachers about training that they received, support they have 
access to, perceptions of the programme, implementation issues and fidelity to the programme. The 
survey results were to be used to assist the selection of a sample selection for the process evaluation. 
However, a decision was made not to proceed with the survey as schools felt strongly overburdened 
by data collection demands as part of this project and further data collection was deemed to be 
unfeasible. (It had already led to two schools dropping out of the process evaluation at a decision- 
making stage.) 


Furthermore, there were no observations of the intervention itself so researchers were unable to see a 
‘Paired Reading’ lesson in practice. The result of this was that researchers had to rely on teachers to 
comment on issues of fidelity or how the programme worked in practice. It is therefore not possible to 
verify the observations and comments of teachers to be an accurate reflection. As such, conclusions 
around the fidelity of implementation should be seen as indicative rather than conclusive. 


Finally, even though specified in the protocol, we unable to carry out separate analyses for pupils with 
EAL due to very low sample sizes. Furthermore, the current report does not include SEN subgroup 
analysis on outcomes of interest. However, the project team will be carrying out further analysis to 
assess the impact of the programme on SEN pupils. 


Interpretation 


The results of this cluster randomised controlled trial do not provide evidence that the Paired Reading 
programme had an impact on pupils’ reading ability. Even more, the findings seem to indicate that the 
programme was in fact detrimental to Year 9 girls (who acted as tutors) levels of reading ability (primary 
outcome) and passage comprehension (secondary outcome that forms a part of the primary outcome). 
This finding is inconsistent with existing research (e.g. Tymms, 2011) that shows overwhelmingly 
positive effects of peer tutoring on raising attainment in school-aged children across a number of 
subjects. 


As indicated above, there is a risk of contamination that trials of educational interventions and treatment 
diffusion — the process by which teacher and pupils in control classes either gain access to the 
intervention being evaluated, or receive a similar intervention by way of being compensated for their 
allocation to control conditions. It is possible that contamination may have reduced the magnitude of 
effect estimates and therefore also increased the chance that estimates are not statistically significant. 


In addition to the risk of treatment diffusion, the process evaluation found that there had been some 
variation amongst participating schools with regards to the set-up, delivery and implementation of the 
programme. For example, it was recommended to run the pupil training in four sessions. However, 
schools used a range of approaches to train the pupils involved, from not giving any advance training 
at all to using the first few sessions to gradually introduce them to the process. Similarly, the length of 
the sessions varied from school to school, with some able to use the recommended half an hour, while 
others were forced to condense the lesson into a shorter time period. There was also a varying level of 
support provided to pupils within the intervention by the teachers involved, based mainly on the reading 
ability of the pupils. However, these appear to be natural variations between the settings of the schools 
involved and are unlikely to have affected the dosage of the intervention for the pupils involved. 


The schools involved in the programme also provided a range of literacy initiatives, in addition to the 
Paired Reading programme. As mentioned previously, these included literacy clubs, support groups 
and withdrawal groups in addition to literacy support and focus within normal timetabled lessons. This 
suggests that the schools involved already placed an emphasis on reading and therefore would have 
been fairly engaged in literacy issues. It is likely that such schools would have given the Paired Reading 
programme their best attempt to make it a success, but that due to the existing emphasis on reading in 
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these school it might be difficult for a programme such as Paired Reading to generate marginal 
improvements in outcomes that might be considered meaningful and thus worthy of investment. 


Furthermore, it has to be acknowledged that ‘business as usual’ in participating schools could be 
accessed by both the intervention group and the control group which makes it somewhat challenging 
to interpret the results. In some of the cases (20 classes) it was the English classes that were replaced 
with Paired Reading, whereas in the majority of cases (32 classes) it replaced classes other than 
English. Therefore, the findings of this study reflect a mixture of providing Paired Reading instead of 
English classes in some of the cases and providing Paired Reading in addition to English classes in 
other cases. 


Future research and publications 


As indicated there is a body of literature showing the positive effects of peer tutoring on pupil outcomes. 
In the light of the current findings one area for future research is to explore the core components of peer 
tutoring interventions that make them work. Furthermore, additional research on the level of technical 
support required to deliver the programme with fidelity would benefit the programme. 
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Appendix A: Parent opt out letter 


: Education Wy Durham 
Endowment University 
Foundation = 


North Tyneside Council 


Peer Tutoring in Secondary Schools 


Helping to raise standards in Literacy 


We are delighted to introduce the Peer Tutoring in Secondary Schools 
programme - an initiative that aims to raise standards in reading through 
peer tutoring. This project is a partnership between North Tyneside 
Council, Queen’s University, Belfast and Durham University. The project is 
funded through a grant from the Education Endowment Foundation. All 
qualifying schools in North Tyneside and some schools in neighbouring 
Authorities are being invited to participate in this project. 


Excellence for All 


The Peer Tutoring in Secondary Schools programme will help to raise reading levels by improving learning. 
It will give teachers new techniques and links well with other developments in schools such as “Literacy 
across the Curriculum” and “Assessment for Learning”. The project will run for three years. 


Peer Tutoring Programme 


During the project your child will undertake peer tutoring in reading. Students in Year 9 will tutor students in 
Year 7, for thirty minutes per session over sixteen weeks in total. During peer tutoring the teacher organises 
materials, manages the class, and monitors the learning that is taking place, to ensure quality. Both tutors 
(Y9) and tutees (Y7) will gain. Such approaches to learning have been proven to raise attainment in schools. 


Why are we doing it? 


° It should help raise levels in reading. 
° Children should be more motivated to learn and more confident in school. 
° It should enhance interpersonal skills. 


Peer Tutoring in Secondary Schools programme 


Paired reading using peer tutoring is a kind of supported reading. It follows a tried and tested method 
which has been refined over many years and was recently used in a very successful project in Scotland. 
Emphasis is placed on reading for understanding and enjoyment. It encourages pupils to talk about the 
text or book and a rapport is built up between the students. Importantly, increases in reading ability bring 
benefits to all areas of the curriculum. 


Your child as part of their class will be randomly selected to either receive the intervention programme and 
assessments, or assessments only. All students will be asked to take pre and post programme student 
assessments. Student test responses and any other pupil data will be treated with the strictest confidence. 


Students will complete the New Group Reading Test which is a computerised assessment, and responses 
will be collected online by the school and accessed by the project research team and the external 
evaluator. Named data will be matched with the National Pupil Database and shared with project research 
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team, external evaluator, EEF, and the UK Data Archive for research purposes. We will not use your 
child’s name or the name of the school in any report arising from the research. 


We expect that your child as part of their class will enjoy doing the tests and being part of the programme. 
Your child may withdraw from the tests at any time. If you prefer for your child NOT to take part, please inform 
their teacher. 


Further information Details 


If you would like to learn more about the Peer Tutoring in Secondary Schools programme, or have any 
questions about any aspect of the initiative, please contact: Maria Cockerill (Project Lead), Peer Tutoring 
in Secondary Schools programme, Early Years and School Improvement Service, Schools, Learning and 
Skills, North Tyneside Council, The Langdale Centre, Langdale Gardens, Howdon, Wallsend, NE28 OHG. 
Email: maria.cockerill@northtyneside.gov.uk 
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Appendix B: Topic guides 


Topic Guide for senior leaders 


Aim of the telephone interview: 


The aim of the interview with teachers is to provide a broad overview of the programme at 
a Strategic level in the school, as well as its perceived impact and implications for the future 


The topic guide: 


This guide sets out a number of necessary contextual and factual topics and questions that 
will be covered during interviews. The guide does not contain follow-up probes and 


questions like ‘why’, ‘when’, and ‘how’, etc. as participants’ contributions will be explored 
in this way, as far as is feasible, during the 30 minute interview. Researchers will use 
prompts and probes in order to understand how and why views, behaviours and 
experiences have arisen. 


The interview will last for approximately 30 minutes. 


1. Introductions 
e Introduce yourself and NatCen Social Research 


e Introduce the study: 
o Evaluation of the Paired Reading Programme 
o Commissioned by the Education Endowment Foundation 
e Overall project aims: 
Oo TOunderstand school experiences of delivering the programme 
Oo To understand the impact (both soft and hard outcomes) that the programme 
is having on students 
Interview is one of 3 interviews with head teachers in different schools 
Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of teachers and schools 
Any questions/concerns? 


Background and Context 


2. Background 


Aim: To gather background information including detail of the context in which the school 
operates, the profile of their pupils and the teacher in question 


e Profile of school: 
o Reading and overall literacy levels (particularly year 7) 
o Turnover of pupils at the school 
o Special Education Needs (SEN) 
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o English as an additional language (EAL) 
o Disadvantaged children (based on FSM take-up) 


The programme itself 


3. Joining the Peer Tutoring Programme 


Aim: To provide understanding of how the school joined the programme, the initial 
expectations and the process 


e First hearing about the programme: 


Oo When was this? 
o Who did you hear about the programme from? 
0 What was the initial understanding? 


e Deciding to join: 


o What did you hope would come out of it for the school? 
o Any apprehensions? 
e Process of joining: 
o Liaising with North Tyneside Council 
o Receiving information and guidance 
o Any improvements to this process 


4. The Peer Tutoring Programme 


Aim: To provide an understanding of the set up and running of the programme from the 
perspective of the leadership team and the school as a whole 


e Process of setting up the programme 

0 Informing staff 

0 Informing pupils 

0 Informing parents 

o Response to the programme from all the above 
e Costs to school 

0 Training teachers 

0 Training pupils 

o Materials 

o Other running costs? 


The future 
5. Impact and implications 


Aim: To understand the wider perceived impact and implications of the programme for 
the school as a whole as well as the potential for rolling the programme out in the future 


e What changed at school as result of programme? 
o Positive changes, e.g. literacy outcomes, improved community feel at 
school, any other? 
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o Any negative changes? 

e Would implement in the future? 
0 If so, would do anything differently? 
0 If not, why not? 


Would recommend to other schools? 
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For Year 7 teachers 


Aim of the telephone interview: 


The aim of the interview with teachers Is to provide a broad overview of the way that tutors 
for the programme are trained, how the paired reading lessons run in practice, what 
barriers there are to successful implementation and what broader impacts the programme 
is having on the students, the department and the school. 


The topic guide: 


This guide sets out a number of necessary contextual and factual topics and questions that 
will be covered during interviews. The guide does not contain follow-up probes and 
questions like ‘why’, ‘when’, and “how’, etc. as participants’ contributions will be explored 
in this way, as far as is feasible, during the 45 minute interview. Researchers will use 
prompts and probes in order to understand how and why views, behaviours and 
experiences have arisen. 


The interview will last for approximately 45 minutes. 


1. Introductions 
e Introduce yourself and NatCen Social Research 


e Introduce the study: 
o Independent Evaluation of the Paired Reading Programme 
o Commissioned by the Education Endowment Foundation 
e Overall project aims: 
o Tounderstand school experiences of delivering the programme 
Oo TO understand the impact (both soft and hard outcomes) that the programme 
is having on students 
Interview is one of 9 interviews with teachers in several different schools 
Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of teachers and schools 
Any questions/concerns? 


Background and Context 


2. Background 


Aim: To gather background information including detail of the context in which the school 
operates, the profile of their pupils and the teacher in question 


e Profile of school: 
o Reading and overall literacy levels (particularly year 7) 
o Turnover of pupils at the school 
o Special Education Needs (SEN) 


o English as an additional language (EAL) 
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o Disadvantaged children (based on FSM take-up) 
e Profile of teacher: 
Oo When qualified 
o Time as teacher 
o Additional responsibilities (e.g. after school clubs etc) 


3. Literacy and reading pre the intervention 


Aim: To get contextual information about literacy and reading in the school, for years 7, 
outside of the Peer Tutoring Programme 


e Literacy pre Peer Tutoring Programme 
0 Literacy as part of normal school syllabus for year 7 
o What interventions in place before/regular format of literacy support 
o What interventions in place now (for pupils not on programme) 
o Nature of these interventions- brief information (e.g. how pupils selected, 
what they go through, who delivers) 


The Peer Tutoring Programme 


4. Training for teachers 


Aim: To provide understanding of how teachers are trained and any preparation that is done 
by the teacher in advance of the programme starting 


e Training for the teaching staff: 


o Nature of this 
o How effective this was 
o How prepared they felt to implement the programme 


e Materials: 


o Which materials used 
Oo How easy these are to use 
o Suggested improvements to these 


Anything they would have liked to have seen added to the training? 
5. Training for pupils 


Aim: To provide an understanding of how pupils are trained to be tutees (Yr 7) and any 
preparation that is done in advance of the programme starting 


e What were Yr7 tutees told about the programme prior to starting 
e Provided any training of any kind? If so... (if not, was any prep needed?) 
e Format of the training sessions for tutees: 


How well these worked 

What guidance was provided to teachers for the training 

Fidelity to the guidance 

Whether tutors and tutees were trained together and how well this worked 


o0o0°0 
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e Materials: (if none, any needed?) 
o Which materials used 
oO How easy these are to use 
o Whether targeted at the right age and abilities 
o Suggested improvements to these 


e Tutees initial attitudes to the programme 
6. Prior to the lesson 
Aim: To understand the process of matching pupils together and selecting pupils to take part 


e Matching pupils 
o Process of matching pupils 
o Use of test results to do this 
o Use of teacher knowledge to do this 
0 Suggested better ways of matching pupils 


Did this work right? Any changes needed? Any suggestions for doing it better? 
7. The lessons 


Aim: To provide understanding of how the lessons run in practice, how consistent they are 
and what barriers there are to successful implementation. 


e Format of the lesson: 
o What did it replace? How easy to timetable and resource? 


o Impact of absences 
o Use of trios or other set-ups 
e Specific aspects of the lesson: 
o Selection of books 
Selection of passages within books 
Correction 
Other specific aspects 
What worked well 
What didn’t work well 


OoOoO000 


e Differences within the lesson between different subgroups (E.g. high ability, low 
ability, middle ability, FSM): 


o How this is managed within the lesson 
0 Differences in fidelity to the programme- was it stuck to by all or varied? (if 
varied, how?) 


e Overseeing tutoring: 


o How inconsistencies are dealt with 

o How consistency could be improved 

o What type of help/guidance needed — for tutors and tutees 
e = Support: 

o Nature of support (from SLT, other teachers etc) 

o Nature of support needed 
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e Barriers to being tutored as perceived by teacher: 
o How these could be tackled 
8. Monitoring pupil progress 


Aim: To establish how pupil progress is monitered both within the lessons (by tutors) and 
more broadly by the department 


e Tutees progress monitoring: 


o Use of pupil record-keeping of tutees 
o How did tutees find this process 


e Teachers monitoring pupil progress: 


o Use of formal assessment 
o Use of informal assessment 
o How involved tutors are in informal assessment 


Impact of the programme and the future 


9. Impacts 


Aim: To provide an understanding of any soft or hard skills gained from the programme, for 
tutors and tutees. To gather information about the positive and negative impacts on the 
maths department and the school more broadly 


e Impacts on tutees: 


Reading ability 

Ability to verbalise thinking 
Impact of immediate feedback 
Confidence 

Social impact 

Enjoyment of reading 


oo0o0o000 


e Impacts on English department and school: 


Any impacts on English department 

Staffing 

Resources 

Time spent planning 

Transferring techniques to broader curriculum 


OoOo000 


10. Views of programme and recommendations 


Aim: To gather views on the programme as a whole, whether they would recommend it and 
how it can be improved. 


e Does the programme meet its aims: 
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0 How/ Why not? 
e Any additional support they would have liked: 


o At what stage? 
oOo In what form? 


e Any improvements: 


Materials 

Support structures 
Programme itself 
Anything else 


oo0o0°0 


e Recommend use to other schools: 
oO Why/ why not? 
11. Sustainability 
e What would they change if the programme was being rolled out nationally? 


e What would be the biggest challenges other schools might face?Will the school 
continue with peer tutoring after this project? Why / why not? 
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For Year 9 teachers 


Aim of the telephone interview: 


The aim of the interview with teachers Is to provide a broad overview of the way that tutors 
for the programme are trained, how the paired reading lessons run in practice, what 
barriers there are to successful implementation and what broader impacts the programme 
is having on the students, the department and the school. 


The topic guide: 


This guide sets out a number of necessary contextual and factual topics and questions that 
will be covered during interviews. The guide does not contain follow-up probes and 
questions like ‘why’, ‘when’, and “how’, etc. as participants’ contributions will be explored 
in this way, as far as is feasible, during the 45 minute interview. Researchers will use 
prompts and probes in order to understand how and why views, behaviours and 
experiences have arisen. 


The interview will last for approximately 45 minutes. 


1. Introductions 
e Introduce yourself and NatCen Social Research 


e Introduce the study: 
o Independent Evaluation of the Paired Reading Programme 
o Commissioned by the Education Endowment Foundation 
e Overall project aims: 
o ToOunderstand school experiences of delivering the programme 
o ToOunderstand the impact (both soft and hard outcomes) that the programme 
is having on students 
Interview is one of 6 interviews with teachers in several different schools 
Digital recording — check OK, and reassure re: confidentiality 
Data kept securely in accordance with Data Protection Act 
How we'll report findings — anonymity of teachers and schools 
Any questions/concerns? 


Background and Context 


2. Background 


Aim: To gather background information including detail of the context in which the 
school operates, the profile of their pupils and the teacher in question 


e Profile of school: 
o Reading and overall literacy levels (particularly year 9) 
o Turnover of pupils at the school 
oO Special Education Needs (SEN) 
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o English as an additional language (EAL) 


o Disadvantaged children (based on FSM take-up) 
e Profile of teacher: 
o When qualified 
Oo Time as teacher 
o Additional responsibilities (e.g. after school clubs etc) 


3. Literacy and reading pre the intervention 


Aim: To get contextual information about literacy and reading in the school, for years 
9, outside of the Peer Tutoring Programme 


e Literacy pre Peer Tutoring Programme 
o What interventions in place before / regular format of literacy support 
o What interventions in place now (for pupils not on programme) 
o Nature of these interventions- brief information (e.g. how pupils selected, 
what they go through, who delivers) 
0 Literacy as part of normal school syllabus for year 9 


The Peer Tutoring Programme 
4. Training for teachers 


Aim: To provide understanding of how teachers are trained and any preparation that is 
done by the teacher in advance of the programme starting 


e Training for the teaching staff: 
o Nature of this 
0 How effective this was 
o How prepared they felt to implement the programme 
e Materials: 
o Which materials used 
Oo How easy these are to use 
0 Suggested improvements to these 
e Anything they would have liked to have seen added to the training? 


5. Training for pupils 


Aim: To provide an understanding of how pupils are trained to be tutors (Yr 9) and any 
preparation that is done in advance of the programme starting 


e What were Yr9 tutors told about the programme prior to starting 
e Provided any training of any kind? If so...(if not, was any prep needed?) 
e Format of the training sessions for tutors: 


o How well these worked 
o What guidance was provided to teachers for the training 
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o Fidelity to the guidance 

o Whether tutors and tutees were trained together and how well this worked 
e Materials: (if none, any needed?) 

Oo Which materials used 

oO How easy these are to use 

o Whether targeted at the right age and abilities 

o Suggested improvements to these 


e Tutors initial attitudes to the programme 
6. Prior to the lesson 


Aim: To understand the process of matching pupils together and selecting pupils to take 
part 


e Matching pupils 


Process of matching pupils 

Use of test results to do this 

Use of teacher knowledge to do this 
Suggested better ways of matching pupils 


o0o0°0 


e Did this work right? Any changes needed? Any suggestions for doing it better? 
7. The lesson 


Aim: To provide understanding of how the lessons run in practice, how consistent they 
are and what barriers there are to successful implementation. 


e Format of the lesson: 
o What did it replace? How easy to timetable and resource? 


o Impact of absences 
o Use of trios or other set-ups 
e Specific aspects of the lesson: 
o Selection of books 
Selection of passages within books 
Correction 
Other specific aspects 
What worked well 
What didn’t work well 


OoOoO000 


e Differences within the lesson between different subgroups (E.g. high ability, low 
ability, middle ability, FSM): 


o How this is managed within the lesson 
0. Differences in fidelity to the programme- was it stuck to by all or varied? (if 
varied, how?) 


e Overseeing tutoring: 


o How inconsistencies are dealt with 
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o How consistency could be improved 

Oo What type of help / guidance needed- for tutors and tutees 
e = Support: 

o Nature of support (from SLT, other teachers etc) 

o Nature of support needed 


e Barriers to tutoring as perceived by teacher: 
o How these could be tackled 
8. Monitoring pupil progress 


Aim: To establish how pupil progress is monitered both within the lessons (by tutors) and 
more broadly by the department 


e Tutors monitoring pupil progress: 


o Use of pupil record-keeping from tutors 
o How easy/difficult tutors find this process 


e Teachers monitoring pupil progress: 


o Use of formal assessment 
o Use of informal assessment 
oO How involved tutors are in informal assessment 


Impact of the programme and the future 


9. Impacts 


Aim: To provide an understanding of any soft or hard skills gained from the programme, 
for tutors and tutees. To gather information about the positive and negative impacts on 
the maths department and the school more broadly 


e Impacts on tutors: 


Reading ability 
Perceptions of own ability 
Confidence 

Attitudes to school 

Social impact 

Enjoyment of reading 


OoOo0oO000 


e Impacts on English department and school: 


Any impacts on English department 

Staffing 

Resources 

Time spent planning 

Transferring techniques to broader curriculum 


OoOoO000 


10. Views of programme and recommendations 
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Aim: To gather views on the programme as a whole, whether they would recommend it 
and how it can be improved. 
e Does the programme meet its aims: 
0 How/ Why not? 
e Any additional support they would have liked: 


o At what stage? 
o In what form? 


e Any improvements: 


Materials 

Support structures 
Programme itself 
Anything else 


o0o0°0 


e Recommend use to other schools: 
oO Why/ why not? 
11. Sustainability 
e What would they change if the programme was being rolled out nationally? 
e What would be the biggest challenges other schools might face? 


e Will the school continue with peer tutoring after this project? 


o Why? 
Oo Why not? 
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Appendix C: Main analysis of the outcomes 


Table C1: Analysis of overall reading ability (primary outcome) - Year 7 pupils - Regression 
output 


(@ersyiie=y ay! Std. Err. 7é P>z 95% Conf. Interval 
0.846 0.018 46.150 0.000 0.810 0.881 
FSM eligibility -0.205 0.649 -0.320 0.752 -1.478 1.067 


Gender 1.050 0.032 0.088 2.012 
Birth month - February -1.616 0.184 -4.004 0.771 


Birth month - March -0.039 0.974 -2.363 2.284 
Birth month - April 1.213 0.317 -1.162 3.587 


Bich month=May 08TH "osm | 3025/1688 
Birth month - June 0.343 1.170 0.290 0.770 -1.951 2.636 


2.867 1.490 1.920 
SSS ee 


EE 
Number of pupils 1300 
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Table C2: Analysis of overall reading ability (primary outcome) - Year 9 pupils - Regression 


output 


Coefficient 


Gender aan 


School 7 


Intercept 


cS) (0 a ol 


95% Conf. Interval 
0.868 0.939 
-1.975 0.602 


-2.306 
-0.990 
-1.173 
-3.047 
-2.394 
-0.492 
-1.270 
0.068 
-1.489 
-2.078 
-1.748 
0.676 1.740 


Number of pupils 1265 
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Table C3: Analysis of passage comprehension (secondary outcome) - Year 7 pupils - 


Regression output 


Coefficient 


Gender 880) 


15.617 


Sehoole 
‘Schools OT 


cS) (0 ian ml 


P>z 95% Conf. Interval 
0.000 0.703 0.780 
0.421 -7.800 3.256 


0.900 -9.460 10.750 
-8.513 12.147 


-15.558 4.949 
(0.814 -8.778 11.176 


-2.503 18.225 
16.065 
12.246 


Number of pupils 1300 
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Table C4: Analysis of passage comprehension (secondary outcome) - Year 9 pupils - 


Regression output 


Coefficient 


Gender 8708 


School 7 


Intercept 


P>z 95% Conf. Interval 
0.000 0.745 0.834 
0.421 -8.797 3.675 


0.902 -12.548 11.068 
—0902| a25a8 | 32.068 


0.707 9.062 
0.267, “5.957 | 21.531 


Number of pupils 1265 
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Table C5: Analysis of sentence completion (Secondary outcome) - Year 7 pupils — Regression 


output 


Coefficient 


Gender 8 


School 7 


Intercept 


P>z 95% Conf. Interval 
0.000 0.698 0.784 
0.244 -7.920 2.014 


-8.677 9.505 
-2.432 16.188 


oar 3572) 14435, 
-5.004 13.632 

10.147 
[ore | 7287 | tose, 
[069 | 2.769 | 35.08 


-9.934 9.081 
-6.013 2.804 
-12.034 7.389 


-8.490 8.495 


0.731 | -10.721 7.523 
(0.609 | -7.404 12.635 
-24.321 -1.293 


-21.040 2.030 
-19.363 0.177 
-9.847 6.208 


-15.755 2.867 
87.422 120.953 


Number of pupils 1304 
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Table C6: Analysis of sentence completion (secondary outcome) - Year 9 pupils — Regression 


output 


Coefficient 


Gender 7288 


School 7 


Intercept 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.786 0.872 


: 
0.000 60.624 97.183 
000 ene2e | 97.308 


Number of pupils 1275 
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Appendix D Subgroup analysis of the outcomes 
Table D1a: Analysis of overall reading ability (primary outcome) - Year 7 pupils (Only FSM) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.04 
(-0.21 to 0.13 ) 


Regression -0.49 -0.60 
ici 0 
SOEHIClenE aC) (-4.33 to 3.34) (-2.76 to 1.60) 


0.801 
ICC (SE) 0.049 0.008 
(0.057) (0.052) 
Variance class level 9.13 0.50 
(SE) (10.70) (3.34) 
Variance pupil level 177.79 63.42 
(SE) (18.25) (6.57) 


Total sample size 246 (58) 246 (58) 
(pupils) 


Table D1b: Analysis of overall reading ability (primary outcome) - Year 7 pupils, FSM 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free 0.068 
school meals and experimental cell (95% Cl) (-2.42 to 2.56) 


ICC (SE) 0.037 
Variance class level (SE) 2.88 
Variance pupil level (SE) 75.32 


Variance component on Free School Meals (SE) 0.00 
(0.00) 


Total sample size (classes) 1,300 (60) 
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Table D2: Analysis of overall reading ability (primary outcome) - Year 7 pupils (Only FSM) - 


Regression output 


Coefficient 


‘Schools aL 
3.794 
School 10 


Number of pupils 


Education Endowment Foundation 


cS) (0 a ol 


95% Conf. Interval 
0.738 
0.319 -2.537 
0.020 0.907 
0.035 10.474 


0.125 -0.911 7.487 
a ee ees 
0.106 0.808) 8,396 


0.555 -5.995 
0.000 
0.505 


3.590 12.331 


15/3 
a 
246 
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Table D3: Analysis of overall reading ability (primary outcome) - Year 7 Interaction model - 


Regression output 


Coefficient 


Gender 3.088 


2.865 
1.014 
0.072 


FSM * Allocation 0.068 


Number of pupils 
Number of classes 


Education Endowment Foundation 


cS) (0 a al 


95% Conf. Interval 
0.810 0.881 
-1.983 1.507 


-4.003 
-2.366 
-1.162 
-1.950 
-0.078 
-2.828 
-1.994 
-1.947 
-2.744 
-1.677 
-3.263 
-0.574 
-0.056 
-2.966 
-1.845 
-0.151 


0.000 10.018 18.703 
a ee ae 
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Table D4a: Analysis of overall reading ability (primary outcome) - Year 9 pupils (Only FSM) 
Unadjusted analysis PNojlUtsiccrolre late Wasi 

-0.06 
(-0.14 to 0.02) 


Effect Size (Cl) 


-2.55 


-0.84 
(-6.94 to 1.84) (-3.13 to 1.44) 


Regression 
coefficient (95% Cl) 


0.255 0.468 
0.081 0.045 
(0.061) (0.070) 
18.20 2.90 
(14.10) (4.57) 
207.07 61.70 
(21.33) (6.74) 
246 (60) 246 (60) 


P-value 
ICC (SE) 


Variance class level 
(SE) 


Variance pupil level 
(SE) 


Total sample size 
(pupils) 


Table D4b: Analysis of overall reading ability (primary outcome) - Year 9 pupils, FSM 


= 
© 
Py) 
° 
. 
} 
=} 
=] 
} 
2 
® 


Adjusted analysis 


Regression coefficient of the interaction between free -0.07 
school meals and experimental cell (95% Cl) (-2.81 to 2.67) 


ICC (SE) 0.016 
Variance class level (SE) 1.28 
Variance pupil level (SE) 78.00 
Variance component on Free School Meals (SE) 3.82 
125,60) 
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Table D5: Analysis of overall reading score (primary outcome) - Year 9 pupils (Only FSM) - 
Regression output 


Coefficient Std. Err. 95% Conf. Interval 


oo | 
m5 [4 
5 [3 
70 | 
so 
s5 | 2 
sea [a 
14 

0.234 -1.829 


Intercept: = 4.506 0.297 -3.956 12.967 
Number of pupils 246 
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Table D6: Analysis of overall reading ability (primary outcome) - Year 9 Interaction model - 


Regression output 


Coefficient 
0.903 
-0.641 
0.134 
1.415 
1.312 
1.661 


FSM eligibility 


Gender 


-0.499 
1.560 
1.341 

-6.220 

-4.273 
2.938 
1.833 


School 7 


School 8 


School 10 


FSM * Allocation 0.073 | 


Number of pupils 
Number of classes 


Education Endowment Foundation 


cS) (0 aa ml 


95% Conf. Interval 
49.640 0.000 0.867 0.939 


-0.640 0.521 -2.599 1.316 
-0.440 0.659 -1.217 0.769 


0.914 -2.308 2.576 
-0.960 3.790 
-1.159 3.783 
-3.054 1.646 
-2.450 2.172 


1.240 0.214 -0.867 3.866 
1.540 0.122 -0.496 4.184 


-1.353 3.496 
0.038 4.833 
-1.570 3.154 
-2.142 0.361 
-1.974 3.134 
1.706 
3.887 
3.848 
-3.292 
-1.340 
5.410 
3.936 


Z P>z 


-0.767 
-1.165 


-7.205 
0.467 
-0.270 


-0.050 0.958 -2.813 2.667 
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Table D7a: Analysis of passage comprehension (secondary outcome) - Year 7 pupils (Only 
FSM) 


Unadjusted analysis Adjusted analysis 
Effect Size (Cl) -0.02 
Regression -0.34 -1.21 
SOC Hiclent 70 CP) (-14.99 to 14.31) (-10.49 to 8.06) 
ICC (SE) 0.028 0.000 
(0.058) (0.00) 


Variance class level 82.03 0.000 
Variance pupil level 2863.09 1208.21 
Total sample size 246 (58) 
(pupils) 


Table D7b: Analysis of passage comprehension (secondary outcome) - Year 7 pupils, FSM 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free 2.46 
i fo} 
school meals and experimental cell (95% Cl) (-8.37 to 13.29) 


0.656 
ICC (SE) 0.036 
Variance class level (SE) 53.48 
Variance pupil level (SE) 1425.47 


Variance component on Free School Meals (SE) 0.000 
(0.000) 


Total sample size (pupils) 1,300 (60) 
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Table D8: Analysis of passage comprehension (secondary outcome) - Year 7 pupils (Only 


FSM) - Regression output 


Coefficient 


Schools HD 
18.208 


School 8 6.098 


15.669 


School 10 39.189 
-1.711 


Number of pupils 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.622 0.780 


: 


9.601 51.683 
0.041 0.945 43.139 


0.042 44.517 


46 
8 
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Table D9: Analysis of passage comprehension (Secondary outcome) - Year 7 Interaction model 


— Regression output 


Coefficient 


Gender 8 


15.529 
10.163 
0.391 


FSM * Allocation 2.458 


Number of pupils 
Number of classes 


Education Endowment Foundation 


Std. Err. 


95% Conf. Interval 
0.703 0.780 


ee 
[oare | anos | 4.28) 
[ora [is oee | 5870 
-9.549 
[eau | 878 | a5 
CS 
eee | as | ae 
-6.924 
i 
20.648 
[eare | aaa | 72277 | 
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Table D10a: Analysis of passage comprehension (secondary outcome) - Year 9 pupils (Only 
FSM) 


Unadjusted analysis PNoplUsiccrolrs late UAci 
Effect Size (Cl) -0.03 
Regression -7.72 -1.88 
egemicient, (9528'S) (-25.50 to 10.07) (-13.61 to 9.86) 
ICC (SE) 0.041 0.058 
(0.056) (0.067) 


Variance class level 172.47 95.56 
Variance pupil level 4011.03 1547.54 
Total sample size 246 (59) 
(pupils) 


Table D10b: Analysis of passage comprehension (secondary outcome) - Year 9 pupils, FSM 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free 4.19 

1 fo} 
school meals and experimental cell (95% Cl) (-9.36 to 17.74) 
ICC (SE) 


Variance class level (SE) 63.05 
Variance pupil level (SE) 1807.60 


Variance component on Free School Meals (SE) 63.05 


(28.03.51) 
Total sample size (pupils) 1,265 (60) 
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Table D11: Analysis of passage comprehension (secondary outcome) - Year 9 pupils (Only 


FSM) - Regression output 


Coefficient 


School 7 


School 8 


School 10 


Number of pupils 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.743 0.926 


: 


-4.827 43.234 
0.192 -7.517 37.426 


0.046 47.830 


-42.300 3.403 


0.154 44.105 


: 


46 
9 
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Table D12: Analysis of passage comprehension (Secondary outcome) - Year 9 Interaction 


model - Regression output 


Coefficient 


Gender 8B 


5.504 
17.064 


FSM * Allocation 4.191 | 


Number of pupils 
Number of classes 


Education Endowment Foundation 


cS) (0 an ml 


P>z 95% Conf. Interval 
0.000 0.745 0.834 


: 
0183/3887 21818 
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Table D13a: Analysis of sentence completion (secondary outcome) - Year 7 pupils (Only FSM) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.15 
Regression -7.67 -7.07 


ICC (SE) 0.068 0.082 
Variance class level 159.90 107.89 
Variance pupil level 2179.63 1213.66 
(pupils) 


Table D13b: Analysis of sentence completion (secondary outcome) - Year 7 pupils, FSM 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free -3.87 
1 fo} 
school meals and experimental cell (95% Cl) (-15.47 to 7.70) 


ICC (SE) 0.009 
(0.011) 


Variance class level (SE) 
Variance pupil level (SE) 1152.17 
(46.85) 


Variance component on Free School Meals (SE) 


Total sample size (pupils) 1,304 (60) 
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Table D14: Analysis of sentence completion (secondary outcome) - Year 7 pupils (Only FSM) - 
Regression output 


Coefficient Std. Err. P>z 95% Conf. Interval 


02/13 
Gender SSR 
5030 
01/9 
iio 
noo 
oa 
650 
soo 


Birth month - November 3.017 . : 0.797 -19.994 26.029 
Birth month - December 7.158 . : 0.519 -14.589 28.906 


-7.068 0.208 -18.070 3.935 
-3. 727 -25.721 17.94 
4.07 707 25.34 
114 444 2.4 
[Schools sa a3 
School 7 -7.261 0.548 -30.949 16.427 
School 8 -13.262 0.374 -42.513 15.989 
-4.401 67 16.1 
School 10 5.808 0.603 -16.051 27.667 
-10.094 44 15. 
Intercept = 112.265 0.000 75.919 | 148.611 
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Table D15: Analysis of sentence completion (Secondary outcome) - Year 7 Interaction model - 


Regression output 


Coefficient 
0.746 
-1.367 
-3.082 
0.297 
6.319 
-3.873 


FSM eligibility 


Gender 


Birth month - December -0.403 
-1.148 
-2.897 
0.595 
-1.862 

2.478 

-12.984 
-8.891 
-9.872 
-1.637 
-5.783 
-3.886 


Allocation 


School 7 


School 8 


School 10 


FSM * Allocation 
Interaction 
Intercept 


Number of pupils 
Number of classes 


102.748 


Education Endowment Foundation 


cS) (0 a ol do Z 
34.460 
-0.330 


P>z 95% Conf. Interval 
0.000 0.703 
0.739 -9.401 


0.788 
6.667 


-0.080 
-0.500 
-0.610 


-0.420 
0.500 


077s | 3988) 


76) 7582 
0238/3888 
052 | 2216 


-8.747 9.342 


-2.964 15.602 


14.838 
9.069 
3.333 
6.423 
8.639 
6.881 


-9.875 
-5.629 


-1.590 


0.113 -19.880 2.098 


-0.420 


0.672 -9.217 5.943 


-0.660 0.511 -15.470 7.698 


a 
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Table D16a: Analysis of sentence completion (secondary outcome) - Year 9 pupils (Only FSM) 


Unadjusted analysis PNoplUsicctolre late UAci s 


Effect Size (Cl) -0.13 
Regression -10.00 -7.17 


ICC (SE) 0.005 0.000 
Variance class level 14.46 0.00 
Variance pupil level 2951.76 1370.10 
(pupils) 


Table D16b: Analysis of sentence completion (secondary outcome) - Year 9 pupils, FSM 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free -8.03 
1 fo} 
school meals and experimental cell (95% Cl) (-19.09 to 3.02) 


ICC (SE) 0.000 
(0.000) 


Variance class level (SE) 
Variance pupil level (SE) 1439.37 
(58.58) 


Variance component on Free School Meals (SE) 


Total sample size (pupils) 1,275 (60) 
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Table D17: Analysis of sentence completion (secondary outcome) - Year 9 pupils (Only FSM) - 


Regression output 


Coefficient 


School 7 


School 8 


School 10 


Education Endowment Foundation 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.702 0.906 
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Table D18: Analysis of sentence completion (secondary outcome) - Year 9 Interaction model - 


Regression output 


Coefficient 


FSM eligibility -2.162 


Gender Tt 


Birth month - December 
‘Schools Baa 


FSM * Allocation -8.032 
Interaction 


Intercept 78.257 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.785 0.872 


0.154 -19.085 3.021 


Number of pupils 1275 
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Appendix E: Subgroup analysis of the outcomes (gender) 
Table E1a: Analysis of overall reading ability (primary outcome) - Year 7 pupils (Only boys) 


Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.05 
(-0.17 to 0.07 ) 


Regression -0.22 -0.77 
icl 9 
edeiicient eae) (-4.37 to 3.94) (-2.50 to 0.97) 


0.919 


ICC (SE) 0.183 0.035 
(0.042) (0.023) 
Variance class level 46.01 3.09 
(SE) (12.23) (2.08) 
Variance pupil level 204.94 86.26 
(SE) (12.06) (5.07) 


Total sample size 638 (60) 635 (60) 
(pupils) 


Table E1b: Analysis of overall reading ability (primary outcome) - Year 7 pupils (Only girls) 


Unadjusted analysis Adjusted analysis 
Effect Size (Cl) 0.02 
Regression -1.38 0.21 
coEMcient sve) (-4.62 to 1.87) (-1.07 to 1.48) 
P-value 0.405 0.751 
ICC (SE) 0.118 0.005 
(0.034) (0.018) 


Variance class level 23.44 0.29 
(SE) (7.45) (1.17) 
Variance pupil level 174.91 64.50 
(SE) (10.04) (3.71) 
Total sample size 667 (60) 665 (60) 
(pupils) 
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Table E1c: Analysis of overall reading ability (primary outcome) - Year 7 pupils, Gender 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free 1.12 
school meals and experimental cell (95% Cl) (-0.80 to 3.04) 


ICC (SE) 0.036 
Variance class level (SE) 2.84 
Variance pupil level (SE) 75.26 


Variance component on Gender (SE) 0.00 
(0.00) 


Total sample size (classes) 1,300 (60) 
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Table E2: Analysis of overall reading ability (primary outcome) - Year 7 pupils (Only boys) -— 
Regression output 


Coefficient Std. Err. 95% Conf. Interval 


CamED 
20 [8 
mE: 
vez |. 
aa 
804 [4 
me: 
-1.173 ; 0.515 2.358 
Ti 
6. 
sa 
567 | 0 
‘School2 -3.122 ; 0.114 0.746 
2 ee 
School5 = 0.566 0.759 4.179 

wa |. 
School 10 0.426 -1.853 
0.724 


Intercept: = 15.502 0.000 9.298 21.705 
Number of pupils 635 
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Table E3: Analysis of overall reading ability (primary outcome) - Year 7 pupils (Only girls) - 
Regression output 


Coefficient cS) (ea la 95% Conf. Interval 


Birth month - February -2.881 -5.974 0.212 


Schools HB 


school 
‘Schools 8288 : 


Intercept 12.439 
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Table E4: Analysis of overall reading ability (primary outcome) - Year 7 Gender interaction 


model —- Regression output 


Coefficient 


Gender 


-3.387 
2.237 
1.119 


Gender * Allocation 14.524 


95% Conf. Interval 
-1.507 1.039 
-0.817 1.845 


-2.365 
-1.178 
-2.981 
-0.036 
-0.515 
-1.999 
-1.925 
Itt 
-2.468 
-3.260 
-0.691 
0.118 4.799 
0.792 3.832 


Number of pupils 1300 
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Table E5a: Analysis of overall reading ability (primary outcome) - Year 9 pupils (Only boys) 


Unadjusted analysis PNoplUtsiccrolre late UW Acl s 


Effect Size (Cl) -0.01 
Regression 1.26 -0.16 


ICC (SE) 0.039 0.006 
(0.023) (0.017) 


Variance class level 10.07 0.50 
Variance pupil level 248.66 90.12 
(pupils) 


Table E5b: Analysis of overall reading ability (primary outcome) - Year 9 pupils (Only girls) 
Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.12 
Regression -4.68 -1.67 
oe 


0.001 0.016 
ICC (SE) 0.047 0.004 
Variance class level 10.28 0.27 
Variance pupil level 208.67 65.36 
(pupils) 
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Table E5c: Analysis of overall reading ability (primary outcome) - Year 9 pupils, Gender 
interaction model 


PNoplUtsiccrolre late UAci 


Regression coefficient of the interaction between free -1.80 
school meals and experimental cell (95% Cl) (-3.79 to 0.18) 


ICC (SE) 0.018 
Variance class level (SE) 1.44 
Variance pupil level (SE) 78.36 


Variance component on Gender (SE) 0.00 
(0.00) 


Total sample size (classes) 1,265 (60) 
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Table E6: Analysis of overall reading ability (primary outcome) - Year 9 pupils (Only boys) — 
Regression output 


Coefficient Std. Err. 95% Conf. Interval 


cas] 3 
906 |= 
wail. 
wo [ 
7a | 
798 | 
azo 
2.039 ; 0.262 5.601 
var. 


Birth month - November 2.945 : : 0.109 -0.661 6.551 
Birth month - December 0.278 : ' 0.875 -3.186 3.741 


m2 
Schoolé | HaaTe 
Schools aid) aT 
73 
ae! 
iereent 882 


Number of pupils 662 
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Table E7: Analysis of overall reading ability (primary outcome) - Year 9 pupils (Only girls) - 
Regression output 


Coefficient Std. Err. 95% Conf. Interval 


Birth month - February 0.284 , -2.896 3.464 


722 
oa 
550 
a! 
sa 
52 
sa 
02 

BO 
Schools OSB 


School 7 -5.417 i . -8.748 -2.087 


School 8 -3.493 -6.924 -0.062 


1.342 


School 10 1.280 -1.100 3.661 


Number of pupils 603 
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Table E8: Analysis of overall reading ability (primary outcome) - Year 9 Gender interaction 


model —- Regression output 


Coefficient 


Gender ort 


1.487 
2.942 


Gender * Allocation — 1.802 | 


Number of pupils 
Number of classes 


Education Endowment Foundation 


Std. Err. 


95% Conf. Interval 
0.865 0.936 
-1.974 0.600 


-2.298 
-1.002 
-1.135 
-3.104 
-2.453 
-0.488 
-1.321 

0.038 
-1.434 
-1.558 
-1.699 
-0.822 
-1.036 
-7.190 

0.484 
-0.260 
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Table E9a: Analysis of passage comprehension (secondary outcome) - Year 7 pupils (Only 
boys) 


Unadjusted analysis Adjusted analysis 
Effect Size (Cl) -0.03 
Regression 0.08 -1.75 
egemicient, (9528'S) (-15.62 to 15.77) (-8.97 to 5.46) 
ICC (SE) 0.170 0.023 
(0.040) (0.021) 


Variance class level 637.50 38.06 
Variance pupil level 3116.69 1650.09 
Total sample size 635 (60) 
(pupils) 


Table E9b: Analysis of passage comprehension (secondary outcome) - Year 7 pupils (Only 
girls) 
Unadjusted analysis Adjusted analysis 


Effect Size (Cl) 0.02 
(-0.1 to 0.14) 


Regression -3.90 0.89 
es: : 
cociticient (2576 Cb) (-16.15 to 8.34) (-4.97 to 6.76) 
0.532 


i a 
ICC (SE) 0.109 0.018 
Variance class level 321.75 
Variance pupil level 2637.41 1180.93 
Total sample size 665 (60) 
(pupils) 
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Table E9c: Analysis of passage comprehension (secondary outcome) - Year 7 pupils, Gender 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free 3.11 
fo} 
school meals and experimental cell (95% Cl) (-5.25 to 11.46) 


ICC (SE) 


Variance class level (SE) 53.14 
Variance pupil level (SE) 1424.28 


Variance component on Gender (SE) 


Total sample size (classes) 1,300 (60) 
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Table E10: Analysis of passage comprehension (secondary outcome) - Year 7 pupils (Only 
boys) —- Regression output 


Coefficient Std. Err. 95% Conf. Interval 


Free School Meals (FSM) 2.912 , -5.788 11.613 


Birth month - February 0.134 ; . 0.987 -15.826 16.094 
Birth month - March -1.135 ; (0.890 | -17.176 14.905 
Birth month - April 4.193 0.591 | -11.118 19.504 


Birth month - May 0.721 ; (0.929 | -15.192 16.634 
Birth month - June 5.418 -10.296 21.133 
Birth month - July 12.985 0,098 -2.413 28.383 


Birth month - September -1.344 : ; -16.753 14.064 
Birth month - October -3.247 (0.675 | -18.431 11.937 
Birth month - November -1.757 (0.824 | -17.203 13.690 
Birth month - December 1.027 0.903 | -15.540 17.593 
-19. ; -35.4 -3.134 
-0. -15. 13.1 
; -9.4 22.762 
School 7 -22.216 -40.186 -4.246 
School 8 -7.428 =27.347 12.491 
-3, -20.51 13.1 
School 10 6.483 -6.430 19.397 
Intercept = 93.210 0.000 69.917 | 116.503 
Number of pupils 635 


Birth month - August 9.485 -6.170 25.140 
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Table E11: Analysis of passage comprehension (secondary outcome) - Year 7 pupils (Only 


girls) - Regression output 


Coefficient 


-7.078 


Birth month - February -9.983 


Birth month - March 4.920 


Birth month - April -3.278 
Birth month - May -10.135 


Birth month - June -3.451 


Birth month - July 0.786 
Birth month - August 2.333 
Birth month - September 6.517 


Birth month - October 8.033 


Birth month - November 0.058 
Birth month - December -2.751 


Allocation 0.892 
12.847 
20.469 


School 7 
School 8 


95% Conf. Interval 
0.721 0.824 


0.000" 
04740800087 
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Table E12: Analysis of overall passage comprehension (Secondary outcome) - Year 7 Gender 


interaction model — Regression output 


Coefficient 


Gender 7.888 


15.590 
10.254 


Gender * Allocation 3.109 | 


Number of pupils 
Number of classes 


Education Endowment Foundation 


P>z 95% Conf. Interval 
0.000 0.703 0.781 
0.405 -7.882 3.180 


-9.468 10.739 
-8.558 12.099 


-15.434 5.081 
0,812 -8.762 11.189 


-2.383 18.353 
-4.038 16.007 


(0.689 | -8.030 12.154 


) 
[osar | -to.ra8 | 6808, 
-9.064 
ad 
20.606 
[acer | aes | 72.06 
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Table E13a: Analysis of passage comprehension (secondary outcome) - Year 9 pupils (Only 
boys) 


Unadjusted analysis Adjusted analysis 
Effect Size (Cl) 0.01 
Regression 4.35 0.30 
egemicient, (9528'S) (-7.78 to 16.49) (-7.99 to 8.59) 
ICC (SE) 0.043 0.026 
(0.023) (0.021) 


Variance class level 182.80 58.51 
Variance pupil level 4176.23 2163.68 
Total sample size 662 (60) 
(pupils) 


Table E13b: Analysis of passage comprehension (secondary outcome) - Year 9 pupils (Only 
girls) 
Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.2 
(-0.214 to -0.186) 


Regression -19.25 -10.89 
ie : 
cociicient (2576 Ch) (-30.05 to -8.45) (-17.48 to -4.30) 


0.000 
ICC (SE) 0.045 0.017 


(0.025) (0.022) 


Variance class level 140.55 

Variance pupil level 2994.40 1380.52 
Total sample size 603 (60) 
(pupils) 
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Table E13c: Analysis of passage comprehension (secondary outcome) - Year 9 pupils, Gender 
interaction model 


PNoplUtsiccrolre late UAcis 


Regression coefficient of the interaction between free -11.94 
school meals and experimental cell (95% Cl) (-21.52 to -2.35) 


ICC (SE) 0.036 
Variance class level (SE) 67.98 
Variance pupil level (SE) 1816.03 


Variance component on Gender (SE) 0.00 
(0.00) 


Total sample size (classes) 1,265 (60) 
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Table E14: Analysis of passage comprehension (secondary outcome) - Year 9 pupils (Only 
boys) —- Regression output 


Coefficient Std. Err. P>z 95% Conf. Interval 


0.772 ; 0.000 0.709 0.835 
Free School Meals (FSM) -5.147 0.299 -14.854 4.560 
Birth month - February -3.835 ‘ : 0.677 -21.905 14.235 


v0 
sr 0 
wr 
oa 


wa 
cone! 
75 
220 
Sehool2 HD 8B 
Sehoolé | aa | B0wS 
Schools nah) Oz28 
J 
a2 
‘Schoo 


22.383 0.011 5.089 39.677 
School 10 15.809 0.613 31.005 


Birth month - August -3.583 ; ; 0.689 -21.134 13.968 
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Table E15: Analysis of passage comprehension (secondary outcome) - Year 9 pupils (Only 


girls) - Regression output 


Coefficient 


Sehoolé aa 
0.535 


School 8 -20.006 


10.511 


School 10 5.882 


Education Endowment Foundation 


P>z 95% Conf. Interval 
0.000 0.762 0.885 
0.873 -7.108 8.368 


2 
0.164 24.490 
2 
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Table E16: Analysis of passage comprehension (secondary outcome) - Year 9 Gender 


interaction model — Regression output 


Coefficient 


Gender 2576 


6.478 
17.411 


Gender * Allocation 11.936 | 


Number of pupils 
Number of classes 


Education Endowment Foundation 


P>z 95% Conf. Interval 
0.000 0.740 0.830 
0.420 -8.781 3.661 


0.345 -5.946 16.987 
0.433 -7.149 16.697 


: 
90/088, 2720 
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Table E17a: Analysis of sentence completion (secondary outcome) - Year 7 pupils (Only boys) 


Unadjusted analysis PNoplUtsiccrolre late UAcl 


Effect Size (Cl) -0.03 
Regression -3.52 -1.56 


ICC (SE) 0.139 0.045 
(0.039) (0.026) 


Variance class level 405.14 65.89 
(SE) (124.87) (39.35) 
Variance pupil level 2509.62 1413.91 
(SE) (147.71) (83.43) 
Total sample size 642 (60) 639 (60) 
(pupils) 


Table E17b: Analysis of sentence completion (secondary outcome) - Year 7 pupils (Only girls) 
Unadjusted analysis Adjusted analysis 


Effect Size (Cl) -0.03 
Regression -5.06 -1.13 
ee 


0.305 0.632 
ICC (SE) 0.111 0.000 
Variance class level 202.16 0.000 
Variance pupil level 1624.20 890.22 
(pupils) 


Education Endowment Foundation 115 


Paired Reading 


Table E17c: Analysis of sentence completion (secondary outcome) - Year 7 pupils, Gender 
interaction model 


PNojiUKs\(cYomr= late lNsiisy 


Regression coefficient of the interaction between free 0.89 
school meals and experimental cell (95% Cl) (-6.63 to 8.41) 


ICC (SE) 0.017 
(0.013) 


Variance class level (SE) 19.67 
Variance pupil level (SE) 1169.21 


Variance component on Gender (SE) 0.000 
(0.000) 


Total sample size (classes) 1,304 (60) 
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Table E18: Analysis of sentence completion (secondary outcome) - Year 7 pupils (Only boys) - 


Regression output 


Coefficient 


Free School Meals (FSM) -4.782 
Birth month - February 4.761 


School OD 
3.440 


School 8 -11.318 


Birth month - August -1.388 


-8.087 
School 10 -0.735 
-4.847 


P>z 95% Conf. Interval 
0.000 0.672 0.797 


0.244 -12.825 3.260 

a ee 
0.530 -10.085 19.607 
0.970 -14.610 15.177 


-4,366 
-22.288 


-30.123 5.903 
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Table E19: Analysis of sentence completion (Secondary outcome) - Year 7 pupils (Only girls) - 


Regression output 


Coefficient 


-1.127 


Birth month - February -9.664 


-2.861 
Schools RTD 
‘Schools a0 a0 
sehools 


-6.279 


Intercept 93.449 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.699 0.816 
0.715 -7.185 4.930 


-6.952 14.761 
-18.011 4.335 


0.999 -10.927 10.912 
0.208 -4.182 19.205 
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Table E20: Analysis of sentence completion (secondary outcome) - Year 7 Gender interaction 


model - Regression output 


Coefficient 


Gender 8S 


2.615 
-9.579 


Gender * Allocation 0.891 | 


Number of pupils 
Number of classes 
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P>z 95% Conf. Interval 
0.000 0.699 0.784 
0.240 -7.952 1.992 


0.361 13.671 


-19.343 0.186 
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Table E21a: Analysis of sentence completion (secondary outcome) - Year 9 pupils (Only boys) 


Unadjusted analysis 


~~ if 
Regression 3.57 
coefficient (95% Cl) (-6.42 to 13.56) 


ICC (SE) 0.018 
(0.020) 


Variance class level 64.53 
Variance pupil level 3526.28 
(pupils) 


Adjusted analysis 


0.04 
(-0.07 to 0.15) 


2.56 
(-3.76 to 8.90) 


0.427 
0.000 
(0.000) 
0.000 
(0.000) 
1653.51 
(90.41) 
669 (60) 


Table E21b: Analysis of sentence completion (secondary outcome) - Year 9 pupils (Only girls) 


Unadjusted analysis 


— es 
Regression -12.71 

ieee r 
coefficient (95% Cl) (-21.45 to -3.97) 


ICC (SE) 0.016 
a ee 

Variance class level 40.25 

Variance pupil level 2527.07 

(pupils) 
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Adjusted analysis 


-0.06 
(-0.18 to 0.06) 
-3.15 
(-8.76 to -2.46) 
0.271 
0.000 
(0.000) 
0.000 
(0.000) 
1177.06 
(67.62) 
606 (60) 
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Table E21c: Analysis of sentence completion (secondary outcome) - Year 9 pupils, Gender 
interaction model 


Adjusted analysis 


Regression coefficient of the interaction between free -7.10 
school meals and experimental cell (95% Cl) (-15.55 to 1.36) 


ICC (SE) 0.000 
(0.000) 


Variance class level (SE) 0.000 
Variance pupil level (SE) 0.000 


Variance component on Gender (SE) 0.000 
(0.000) 


Total sample size (classes) 1,275 (60) 
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Table E22: Analysis of sentence completion (secondary outcome) - Year 9 pupils (Only boys) - 
Regression output 


Coefficient Std. Err. P>z 95% Conf. Interval 


000 25 
27/9 
wai 0 
wea 9 
72/0 
SoHo 
oa 0 
at 
sao 


Birth month - November 7.876 : : 0.314 -7.446 23.199 
Birth month - December 8.052 . : 0.286 -6.746 22.850 


2.560 0.427 -3.760 8.879 
‘School2 3.048 0.666 -10.808 16.905 
School4 9.592 0.114 -2.292 21.477 
School5 0.154 0.982 -13.272 13.580 

-0.206 0.977 -13.950 13.539 


School 7 -17.815 -33.553 -2.077 


School 8 -10.288 


erent HD 


Number of pupils 662 
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Table E23: Analysis of sentence completion (secondary outcome) - Year 9 pupils (Only girls) - 


Regression output 


Coefficient 


Sehoole OBL 
5.848 


School 8 0.196 


-4.360 


School 10 3.859 


Education Endowment Foundation 


cS) (0 a ol 


P>z 95% Conf. Interval 
0.000 0.785 0.915 


-22.737 4.725 


-16.120 7.399 


: 
0.000 40.746 95.373 
000 90745 | 95373 
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Table E24: Analysis of sentence completion (secondary outcome) - Year 9 Gender interaction 


model - Regression output 


Coefficient 


Gender a 785 


2.595 
1.812 


Gender * Allocation — -7.095 | 


Number of pupils 
Number of classes 


Education Endowment Foundation 


Std. Err. 


P>z 95% Conf. Interval 
0.000 0.783 0.870 


0.197 13.179 
10.841 
05) aasa ser 
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Appendix F: Security classification of trial findings 


2. Power: 5. Threats to 
1.Design: ee 3. Attrition: 4. Balance: ST TT 
h EERSTE ee ae ty: 
What is the WY ae What is the Adjustment veld 


: minimum Adjustment to 
quality of the (AV/-) Ie) ite | go) to rating to 
(ol- ¢-tot ¢-] 0) (= account for issues 


(ofS 4g Mo) mda out from the account for 


Liation? effect at the liation? = = with 
evaluation? evaluation? alance? 
start? interpretation? 
Evaluation design Implementation Analysis and interpretation 


Rating 1. Design 5 3. Attrition 4. Balance oP Mal acyc\ com ce) 
WEN Celia 


am No threats to validity 
Fair and clear experimental 
design (RCT, RDD) < 0.3 < 20% 


Well-matched comparison 
(quasi-experiment) < 0.4 < 30% 
Matched comparison 
(quasi-experiment) < 40% 
oor or no matchin 
p g y y 
No comparator Imbalanced on er 


The final security rating for this trial is 4 @. This means that the conclusions have moderate to high 
security. 


This evaluation was designed as a randomised controlled trail. The sample size was designed to 
detect a MDES of less than 0.2, by design. At the unit of randomisation (class), there was zero 
attrition, and extremely low attrition at the pupil level also. Some of the baseline measures of 
attainment were a little imbalanced at baseline (at the level of ES=0.1), but since many measures 
were compared and attrition was low it is likely that these small differences arose by chance. The 
post-tests were administered by the schools by teachers who were aware of the treatment allocation. 
As there was no independent invigilation (of even a sample of schools), the padlock rating is reduced 
to4 @. 
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Appendix G: Cost rating 


level mclilale) Description 


£ Very low: less than £80 per pupil per year. 
EEL Low: up to about £200 per pupil per year. 


EEE Moderate: up to about £700 per pupil per 
year. 


ELEEE High: up to £1,200 per pupil per year. 


ELEEEE Very high: over £1,200 per pupil per year. 
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